Augmented reality system

ABSTRACT

Systems and methods are disclosed for recommending products or services by receiving a three-dimensional (3D) model of one or more products; performing motion tracking and understanding an environment with points or planes using accelerometer sensor and estimating light or color in the environment using one video camera without a depth sensor in a mobile phone; acquiring sensor data from sensors and optimizing features extracted from each image and sensor data, where a feature conveys data unique to the image at a specific pixel location; and projecting the product in the environment.

The present invention provides a method for fitting products.

BACKGROUND

In fields such as surgery, clothing, footwear, and 3D printing, amongothers, needs exists for a method to capture the anatomy in the worldand either reproduce them with a 3D printer, share, or virtually operateon the anatomical 3D model. Existing system to scan the user in threedimensions require specialized hardware. For example, some systems cancreate an estimate of the depth of a body by simultaneously acquiringimages with multiple image capture devices, by using a known andstructured light source, by using a laser solution, or some combinationthereof. This creates additional expenses and software requirements forthe user.

SUMMARY

In one aspect, systems and methods are disclosed for recommendingproducts or services by receiving a three-dimensional (3D) model of oneor more products; performing motion tracking and understanding anenvironment with points or planes using accelerometer sensor andestimating light or color in the environment using one video camerawithout a depth sensor in a mobile phone; acquiring sensor data fromsensors and optimizing features extracted from each image and sensordata, where a feature conveys data unique to the image at a specificpixel location; and projecting the product in the environment.

In another aspect, systems and methods are disclosed for recommendingproducts or services by receiving a 3D model of a product; capturing areference object with a predetermined dimension in an environment wherethe product is to be placed using a mobile camera; determining one moredimensions of the environment relative to the predetermined dimension ofthe reference object; scaling the 3D model of the product based ondimensions of the environment and the product; and generating anaugmented or virtual reality display of the product in the environment.

In implementations, the reference object can be a coin or a sheet ofpaper with predetermined dimensions. The mobile camera can be a smartphone or a portable camera. The camera can be an infrared camera. Theproduct can be an appliance or furniture, a wearable item, a jean, or ashirt. For clothing, the system can render an image of the object on amannequin. The system can monitor user health by analyzing changes inthe 3D model over time. The system can analyze a user anatomical portionand selecting a best fit from apparel variations. The product can becosmetic product, a facial makeup product, or a hair product. The systemincludes motion tracking, area learning and depth sensing the product.The system can create a 3D model using infrared images. The systemincludes identifying one or more best fitting products to theenvironment and displaying recommendations with one or more best fittingproducts in the environment. The best fitting products can be clothing,shoes, cosmetics, appliances or furniture. The method includes capturing3D model of user's feet; identify the subject's current best fittingshoe products; set each best fitting shoe product's inside dimensionwith dimensions from the 3D model plus a predetermined gap; correlatingdifferent manufacturer's shoe sizes and creating correspondences amongdifferent manufacturer shoe products; and recommending a new shoe forthe subject by looking up the correspondences among differentmanufacturer shoe products.

In another aspect, a method for best fitting product variations to anenvironment by receiving a 3D model of a product with one or moreproduct variations; capturing a reference object with a predetermineddimension in an environment where the product is to be placed using amobile camera; determining one more dimensions of the environmentrelative to the predetermined dimension of the reference object; scalingthe 3D model of each product variation based on dimensions of theenvironment and characteristics of the product variation; and generatingan augmented or virtual reality display of the product in theenvironment.

In yet another aspect, a method for recommending a service includesreceiving a model of a service to be applied to a target object;capturing a reference object with a predetermined dimension in anenvironment where the service is to be applied to the target objectusing a mobile camera; determining one more dimensions of theenvironment relative to the predetermined dimension of the referenceobject; generating a 3D model of the service as applied to the targetobject; scaling the 3D model of the generated 3D model based ondimensions of the environment and the product; and generating anaugmented or virtual reality display of the product in the environment.

In implementations, the service to a product can be for one of: acosmetic product, a plastic surgery medical device, a facial makeupproduct, a hair product. For example, for make up, the method includescapturing images of a face and a reference object from a plurality ofangles using a mobile camera; creating a 3D model of the face from theimages with dimensions based on dimensions of the reference object;selecting a makeup pattern or color from a plurality of makeup productvariations; and blending the makeup pattern or color onto the 3D model;and displaying the makeup color on the face. If the target object is abreast implant, the method includes recommending a breast augmentationsizing to a patient. In another aspect, a camera tracks movements and a3-D scanner analyzes the viewer's physique. Body recognition softwareanalyzes the body shape to determine weight loss or gain. In addition toshoe/clothing suggestions, the system can provide clothing/jewelry/hairstyling suggestions along with augmented reality view of the suggestionsso that the user can visualize the impact of the clothing or jewelry orstyling. Facial recognition software inspects the face shape todetermine health. The smart mirror can provide make-up suggestions alongwith augmented reality view of the applied suggestions so that the usercan visualize the impact of the makeup. The smart mirror can providenon-surgical body augmentation suggestions such as breast/buttockaugmentations along with augmented reality view of the body enlargementsor size reduction so that the user can visualize the impact of thefootwear or apparel when worn, along with body enhancement, clothing orjewelry or hair styling changes. In yet another aspect, built-in sensorsin combination with mobile phone usage pattern and social networkcommunications can detect signs of stress and other mental/emotionalhealth states of the user. The smart insole or shoes with sensors couldalso be combined with other health-related apps to keep track of caloriecount, vital signs, fitness level and sleep quality. By extrapolatingfrom the user's current behaviors, vitals and bone and muscle structure,the augmented-reality mirror can forecast the user's future health. Thecamera can measure breathing activity and/or heart rate of the user infront of the mirror or alternatively the system can bounce WiFi off thechest to detect breathing activity. The mirror highlights hard-to-seechanges in the body, such as increased fatigue, minute metabolicimbalances and more. A DNA analyzer can receive swipes from tongue, ear,and saliva, bodily fluids to capture genetic data at a high frequencyand such data can be correlated with the fitness wearable devices forsigns of health problems. Additionally, the data can be analyzed at ametropolitan level for public health purposes.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1A shows an exemplary process for creating a 3D model of a bodyportion such as a foot.

FIG. 1B shows exemplary anatomical points on the foot.

FIG. 1C shows shoe matching based on key sections and girths of thefoot.

FIG. 1D shows an exemplary process for determining back foot features.

FIG. 1E shows an exemplary shoe scanner for scanning interior dimensionsof shoes/footwear.

FIG. 1F shows exemplary footwear with sensors and heater/cooler embeddedtherein.

FIG. 1G shows exemplary templates for rapid, inexpensive 3D printing ofinsoles with sand.

FIGS. 1H-1I show exemplary systems and techniques for manufacturing involume with a wide range of materials are disclosed for fabricatingshoes at mass customization scale.

FIGS. 1J, 1K, 1L, and 1M show a first container embodiment, a mastershape and a vacuum cap, and further show a sequence of operations tocreate a shaped impression, complementary to the master footwear shape,in the surface of one elastomeric membrane face of the container.

FIG. 1N shows exemplary composite insoles with shock absorbing springsat the bottom.

FIG. 1O shows exemplary orthotic insole or sole produced using the abovesystem.

FIG. 2 shows another exemplary process for creating a 3D model of thebody.

FIG. 3A shows an exemplary user interface on the phone for virtuallytesting the make-up products prior to order, while FIG. 3B shows anexemplary method for testing make up techniques and/or products.

FIGS. 4A-41 illustrate representations of representing eight of theclassic facial configurations.

FIGS. 5A-5E illustrate various computer recommendation oneye-formations.

FIG. 6A-6B illustrate computer applications of selected cosmeticcompositions such as dark foundation, eye disguise or highlighterapplied to the virtual face.

FIGS. 7A through 7I are representative illustrations of the varioustypes of lip outlines of lip configurations for makeup on the virtualface or model of the user.

FIG. 8A shows an exemplary process for suggesting styles for the user.

FIG. 8B shows an exemplary process for producing custom fashion clothingbased on images or videos of a model or a celebrity that the user likes.

FIG. 8C shows a mass-customized clothing fabrication network that isdriven by current fashion and hot celebrity trends.

FIG. 9A shows an exemplary hair style suggestion process.

FIG. 9B shows an exemplary furniture or appliance suggestion process,and FIG. 9C shows UI.

FIG. 10 shows an exemplary medical cosmetic suggestion process with the3D body model.

FIG. 11 shows an exemplary system for receiving health information fromthe user body and for mining health data as part of precision medicine.

DESCRIPTION

As used in this document, the singular forms “a,” “an,” and “the”include plural references unless the context clearly dictates otherwise.Unless defined otherwise, all technical and scientific terms used hereinhave the same meanings as commonly understood by one of ordinary skillin the art. As used in this document, the term “comprising” means“including, but not limited to.”

Deformable 3D Master Models of Body Portions

First, the process forms a set of deformable models of the body such asthe face, hand, or foot models. The deformable 3D master model issubsequently match to the user's body simply by matching referencepoints. In various embodiments, the deformable models can be done bysex, weight or by disease classification, for example.

In one embodiment for foot products, an initial set of feet is scannedusing a smart phone, a camera, or a combination of camera and infraredlaser/camera, and the points are used in constructing a library ofdeformable 3D model of feet. The library of deformable 3D models arecreated and matched to images of the feet through deformations to thepoints of interest to precisely match the user's actual feet. Aphysically accurate 3D model can be created therefrom withoutsignificant storage of the 3D points and the 3D model of the user can bedone using minimal computation resources. The 3D models of feet allownew feet to be customized to a particular 3D template, thus avoidingstorage requirements of millions or billions of 3D models of feet. Toenhance accuracy, instead of one model for everyone, a plurality ofdeformable master models can be formed for each of discrete sizes suchas US sizes 4, 4.5, 5, 5.5, 6, 6.5, 7, 7.5, 8, 8.5, 9, 9.5, 10, 10.5,11, 11.5, 12, 12.5, 13, 13.5, 14, 14.5, 15, 15.5, 16, 16.5, 17, 17.5,and 18, for example.

The morphable or deformable foot model can be generated from a set ofunregistered 3D foot model. The process computes a dense point-to-pointcorrespondence between the vertices of the foot. The process finds thebest match of a given foot only within the range of the morphable model.To determine residual deviations between a new foot and the best matchwithin the model, as well as to set unregistered prototypes incorrespondence, in one embodiment the process uses an optic flow methodthat computes correspondence between two feet without the need of amorphable model.

Constructing a deformable or morphable foot model from a set ofunregistered 3D scans is done through a computation of the flow fieldsbetween each foot and an arbitrary reference foot. Given a definition ofshape and texture vectors for the reference face to each face in thedatabase can be obtained by means of the point-to-point correspondenceprovided by feet in the database.

A recursive process for enhancing the deformable foot model is describednext. The process finds rough correspondences to the new foot using the(inadequate) morphable model and then improves these correspondences byusing an optic flow method. Starting from an arbitrary foot as atemporary reference, preliminary correspondence between all other feetand this reference is computed using the optic flow algorithm. On thebasis of these correspondences, shape and texture vectors can becomputed. To handle noise, each deformable model is run through aplurality of foot scans, and if the same correspondences appear, theyare used and correspondences from noisy data are discarded.Alternatively, their average serves as a new reference foot. The firstmorphable model is then formed by the most significant components asprovided by a PCA decomposition. The current morphable model is nowmatched to each of the 3D feet according and the optic flow methodcomputes correspondence between the 3D foot and the approximationprovided by the morphable model. Combined with the correspondenceimplied by the matched model, this defines a new correspondence betweenthe reference foot and the example. Iterating this procedure withincreasing expressive power of the model (by increasing the number ofprincipal components) leads to reliable correspondences between thereference foot and the examples, and finally to a complete morphablefoot model.

Another embodiment determines a 3D Correspondence using Optic Flow andfinds corresponding points in grey-level images and a gradient-basedoptic flow method to establish correspondence between a pair of 3Dscans, taking into account color and radius values simultaneously. Thealgorithm computes a flow field that minimizes differences of in a normthat weights variations in texture and shape equally. Surface propertiesfrom differential geometry, such as mean curvature, may be used asadditional components. On foot regions with little structure in textureand shape, the results of the optic flow method are sometimes spurious.A smooth interpolation is done based on simulated relaxation of a systemof flow vectors that are coupled with their neighbors. The quadraticcoupling potential is equal for all flow vectors. On high-contrastareas, components of flow vectors orthogonal to edges are bound to theresult of the previous optic flow computation. The system is otherwisefree to take on a smooth minimum-energy arrangement. Unlike simplefiltering routines, the technique fully retains matching qualitywherever the flow field is reliable. Optic flow and smooth interpolationare computed on several consecutive levels of resolution.

In one embodiment, the closest deformable model is selected to match toa new foot. The selection of the closest model can be done based on sizeor a suitable attribute such as weight or disease, for example. Theclosest deformable model is then selected to be matched to the new foot.The method transfers foot motion vectors from a source foot model to atarget user model having different geometric proportions and meshstructure (vertex number and connectivity). Using an automated heuristiccorrespondence search, the system can select fewer than ten points inthe model to match to a new scan. The method allows new 3D foot scans tobe easily retargeted to a library of foot templates or models.

A similar process can be used to create deformable face models, handmodels, stomach models, ear models, breast models, and buttock modelsfor cosmetic and health purposes.

3D Models of Subjects Using Phone Cameras

FIG. 1A shows an exemplary process for creating a 3D model of a bodyportion such as a foot. First, the body portion such as a foot is placedadjacent an object (coin or grid) with known dimensions (20). Forexample, an A4 sheet of paper can provide known dimensions to scale tothe foot. Similarly, a quarter coin can provide known thickness anddiameter dimensions used to scale the image. Alternatively, paper withgrids imprinted thereon can be used to provide reference dimensions.

The system then takes multiple images or videos of the foot and object(22). In one embodiment, the user can simply use a smart phone directlycreate the 3D model in the phone or upload the images to a server forprocessing. Next, the system can determine dimensions of points ofinterest on the foot based on object size (24). For example, FIG. 1Bshows exemplary points of interest including anatomical Points: (1)Rearest point of the Heel (2) Most advenced point of the 2nd Toe (3)Point of the Instep (4) Insertion of Achille's tendon in Calcaneus (5)Most prominent point of the internal malleolus (6) Most prominent pointof the external malleolus (7) Below internal malleolus (8) Belowexternal malleolus (9) Most prominent point of the external heel (10)Most prominent point of the internal heel (11) Most prominent point ofthe head of the 1st metatarsal (12) Highest point of the 1st toe (13)Most lateral point of the 5th toe (14) Most prominent point of the 5thmetatarsal (15) Styloid-apophysis of the 5th metatarsal (16) Lowestpoint of Navicular (17) Forest point of the 5th toe.

Turning back to FIG. 1A, the process selects a standard foot template(26) and morph/warps the standard foot template to match points ofinterest (28), and then selects a footwear or a shoe with interior bestmatching the morphed foot template (30). For example, in FIG. 1C, thematching to the footwear or shoe can be based on key sections and girthsof the foot: (1A) Toe Section (2A) Metatarsals Section (3A) MidfootSection (4A) Heel Section (5A) Profile of the foot (6A) plantar contour.“Footwear” refers to any type of apparel that may be worn on a person'slower body, specifically the feet and optionally also the lower legs.Examples include athletic shoes and other shoes, work boots, ski bootsand other boots, sandals, slippers, and any other apparel item designedto be worn on the foot and optionally also the lower leg.

The process can store body dimensions in a computer, a data storagedevice, cloud storage, or in a separate data storage facility as well asidentifying information for a plurality of wearable items and datarelated to each wearable item. The data can include a set of internalmeasurements and other fit and performance parameters that may beobtained for each wearable item and imported into the data set such thata two dimensional (2D) or three dimensional (3D) representation of thewearable item may be constructed. The data set also may include feedbackabout the wearable items as reviewed by multiple consumers, as will bedescribed in more detail below. For example, for a footwear model, theinternal measurements can include a total length measurement, a totalwidth measurement, heel width, arch length and arch width. Whenapplicable, additional parameter measurements can also be stored,including, but not limited to, toe box height, forefoot height, and archheight. Three dimensional measurements may be stored within the data setas well, such as toe box girth, forefoot girth, and heel to toe girth.Measurement parameters such as tapering or change in width as apercentage of total length can also be stored within the data set. Itshould be noted that this list of measured parameters is provided by wayof example only, and additional parameters measurements may be includedsuch as heel height, arch height, girth, foot opening diameter, and anyother relevant information. In additional to dimensional measurementsdescribed above, other parameter measurements may be associated with afootwear model depending on model type. For example, a miming shoe mayhave feature-based parameter measurements associated with stabilitywhether or not the shoe has motion control, racing spikes, and any otherrelevant parameters. Tactile measurements such as cushioning, stretchand deformation also may be available for various areas in the footwearmodel. The system may receive these parameter measurements from one ormore scanning devices that scan the footwear model and collectmeasurement data from the footwear model.

In one embodiment, the user captures images from all angles around thebody (such as the foot). When the user finishes scanning the foot, theplurality of images is transferred to a processor. For exemplarypurposes, the processor may be on a remote server. A reconstruction andgeneration operations are performed on the plurality of images. Anoptimization is performed on the plurality of images to simultaneouslydetermine a pose of the image capture device for each image in theplurality of images, as well as a camera matrix for the image capturedevice used, the camera calibration matrix or camera intrinsics, as wellas one or more radial distortion parameters. The pose of the imagecapture device includes an X, Y, and Z location in a universalcoordinate frame, which describes distances from an origin in a threedimensional coordinate system along three orthogonal basis vectors. Thepose of the image capture device also includes a roll, a pitch, and ayaw, which correspond to rigid body rotations about each of the threeorthogonal basis vectors. The total pose of the image capture device maybe described as <x, y, z, r, p, q>, or may also be given as atranslation in three dimensions plus a quaternion, or a rotation matrixand translation vector. The camera matrix includes a two-dimensionalcenter point, a focal length in a first axis, and a focal length in asecond axis. In addition, one or more radial distortion factors whichdescribes a radial distortion associated with the plurality of imagesdue to a lens used in the image capture device is extracted. As analternative to a single radial distortion factor expressing, for examplea fish-eye lens, a series of coefficients may be extracted whichexpresses additional radial distortion parameters if the lens model is apolymer. For exemplary purposes, the optimization is a non-linear leastsquares optimization using a series of points associated with theregular pattern of first shapes and second shapes as determined in everyimage. In an alternative embodiment, the camera can be equipped with aplurality of sensors. The sensors may include accelerometers, sonar,gyroscopes, magnetometers, laser range finder, and global positioningsystems where the surface with the regular pattern of first shapes andsecond shapes is not required. In the scanning step, sensor data fromthe plurality of sensors is also acquired between every image capturedin the plurality of images. The sensor data is also sent to theprocessor in the scanning step. In the reconstruction, the optimizationis performed not over the series of points associated with the regularpattern of first and second shapes, but features extracted from eachimage in the plurality of images, as well as the sensor data. A featureconveys data which is unique to the image at a specific pixel location,such as unique image gradients or pixel intensities. For exemplarypurposes, features are extracted from the images and can be related to(1) Rearest point of the Heel (2) Most advenced point of the 2nd Toe (3)Point of the Instep (4) Insertion of Achille's tendon in Calcaneus (5)Most prominent point of the internal malleolus (6) Most prominent pointof the external malleolus (7) Below internal malleolus (8) Belowexternal malleolus (9) Most prominent point of the external heel (10)Most prominent point of the internal heel (11) Most prominent point ofthe head of the 1st metatarsal (12) Highest point of the 1st toe (13)Most lateral point of the 5th toe (14) Most prominent point of the 5thmetatarsal (15) Styloid-apophysis of the 5th metatarsal (16) Lowestpoint of Navicular (17) Forest point of the 5th toe. The features can beunique image gradients or pixel intensities or can also be mathderivatives using Harris corners, FAST features, FREAK features, SIFTfeatures, ORB features, SURF features, BRISK features, or the like.Codebook of features can be used to map the user anatomy to one of aplurality of deformable or morphable 3D foot models. The process canthen selects footwear or a shoe with interior best matching thedeformable/morphable foot template key sections and girths of thefootwear: (1A) Toe Section (2A) Metatarsals Section (3A) Midfoot Section(4A) Heel Section (5A) Profile of the foot (6A) plantar contour.

Next, the process can form codebooks of foot features. Alternatively, acontinuous density model can be used because it avoids any errors thatcould be introduced in the quantization phase. The codebook approachclassifies each frame into one of N categories, each represented bycanonical vector that is associated with a symbol in the code book. Oncea foot is classified by a codebook, can be represented by a singlesymbol that indicates which code value it is closest to. Code bookvectors are defined by training on a training corpus of feet thatminimizes the overall distortion, which is the sum of each inputvectors' distance from the code book vector that it is identified with.The process iteratively improves the entire set of code book vectorsusing a K-means clustering algorithm where, given an initial set of Ncode book vectors Ci and a set of training vectors, the processperforms:

Classification: Cluster the training vectors by its closest code bookvector according to a distance function;

Centroid Update: Update each code book vector to be the centroid(relative to the a difference function used) of the training vectorsassigned to it.

Iteration: If the improvement in overall distortion is greater than athreshold, then repeat from step 1.

Continuing in the reconstruction, once the pose of the image capturedevice and the camera matrix is determined for every image in theplurality of images, it is possible to estimate the depth at specificimages in the plurality of images using both intensity values containedin the image as well as the image capture device pose for every image.For exemplary purposes, the depth may be acquired by a minimization ofan energy defined. The minimum of the energy may be solved for byperforming a Legendre-Fenchel transform and expressing the optimizationin both a primal and dual variable. By expressing the problem in boththe primal and dual forms, it is possible to use a primal-dual hybridgradient approach to finding the minimum of the energy. Because aprimal-dual hybrid gradient is used, the minimum may be determined byperforming a primal descent and a dual ascent for every pixel in theimage in parallel on a graphics processing unit (GPU). Sequentialsubsets of the plurality of images are used to form a depth image,wherever a depth image is desired, by first determining the inversedepth for every pixel in the cost volume which maps to the lowest cost.Once a minimum is estimated, a dual ascent is performed in dual step, aprimal ascent is performed in a primal step and an update is performedin an update step. In the update step, a similar search through the costvolume is performed as in the depth estimate, however the search isaugmented by the difference of the primal variable with the slackvariable, squared, divided by twice the mediation variable. The dualstep, primal step, and update step are repeated until a stoppingcriterion is reached. For exemplary purposes, the stopping criterion isreached once the mediation variable is reduced below a threshold or achange in the energy computed is below a certain threshold. Once thestopping criterion is reached, the depth at every pixel calculated isstored in a depth image.

Alternatively, a buffer of a predetermine number of frames is createdfrom a video sequence. If a previous depth image is known (determinedvia the previous depth estimate or by raycasting a truncated signeddistance function storing a fusion of previous depth estimates), thefull pose of the image capture device for every image is updated byperforming dense tracking using the previous depth estimate, image takenat the previous depth estimate, and a current image. Dense trackingcalculates the pose by performing a minimization with respect to thepose of a reprojection error between the previous image, the previousdepth, and the current image using every pixel of both images.

Once the cost volume is calculated, a depth per frame is calculated byfirst performing a minimum search along every inverse depth element forevery pixel in the cost volume. This rough depth estimate is thensmoothed using a weighted Huber regularizer via the same primal-dualhybrid gradient optimization schema as above. To further increase theaccuracy of the depth estimates, the output of the optimization is usedto initialize a wide baseline polishing step. In this wide baselinepolishing step, a linearization of reprojection errors from thereference image of four additional frames further from the referenceframe than the 20 frame selected subset, but within 80 cm of thereference frame, is regularized with a similar weighted Huberregularizer and minimized using a primal-dual hybrid gradient approachyielding a depth image. All of the depth images form a series of depthimages. Since the pose of the device is known relative to the surface,it is possible to remove all information from the depth image that is ator below the surface. This leaves only the foot object in an updateddepth image. The series of updated depth images may be stored in avolumetric representation of depth. For exemplary purposes, thevolumetric representation of depth is a signed distance function. Eachdepth image is then loaded into the signed distance functionrepresentation. A model is formed using the volumetric representation ofdepth and stored in a model file. For exemplary purposes, the model fileis a mesh. Further, it is contemplated that the model file is createdfrom the volumetric representation of depth. One such volumetricrepresentation of depth is a signed distance function. Alternatively, atruncated signed distance function may be used. Once every image isacquired, it is fused into the signed distance function. The model filemay be extracted from a signed distance functions by such algorithms asmarching cubes, marching tetrahedral, or Poisson reconstructions.

Although the foregoing discusses phone based cameras, other consumercameras that work with a desktop computer can be used. In one embodimentthe Microsoft Kinect camera can be used, while in another embodiment, a3D camera such as the Intel RealSense uses three components: aconventional camera, a near infrared image sensor and an infrared laserprojector. Infrared parts are used to calculate the distance betweenobjects, but also to separate objects on different planes. In oneembodiment, a processor to translate the edges as mouse movement andmouse clicks to control the vehicle by moving hands. They serve forfacial recognition as well as gestures tracking. The Intel 3D camera canscan the environment from 0.2 m to 1.2 m. Its lens has a built in IR cutfilter. The video camera has a frame rate up to 60 fps with a 90° FOV,moreover its lens has an IR Band Pass filter. The IR laser integrates aninfrared laser diode, low power class 1, and a resonant micro-mirror.The 3D camera can provide skeletal and depth tracking and may gatherspatial data that describes objects located in the physical environmentexternal to the depth sensor (e.g., the user's bath room). The skeletaland depth tracking technology may be implemented in a depth sensor(e.g., the Kinect, the Intel Realsense), stereo cameras, mobile devices,and any other device that may capture depth data. In some exampleembodiments, the skeletal and depth tracking technology is implementedon a server using algorithms that utilize the RGB and depth channels. Insome example embodiments, depth sensing technologies use structuredlight or time of flight based sensing. For example, an infrared(hereinafter, also “IR”) emitter that is part of the preference analysismachine 310 and that is located in the user's living room, may project(e.g., emit or spray out) beams of infrared light into surroundingspace. The projected beams of IR light may hit and reflect off objectsthat are located in their path (e.g., the user or a physical object inthe user's living room). A depth sensor (e.g., located in the user'sliving room) may capture (e.g., receive) spatial data about thesurroundings of the depth sensor based on the reflected beams of IRlight. In some example embodiments, the captured spatial data may beused to create (e.g., represent, model, or define) a 3D field of viewthat may be displayed on a screen (e.g., of a TV set, computer, ormobile device). Examples of such spatial data include the location andshape of the objects within the room where the spatial sensor islocated. In some example embodiments, based on measuring how long ittakes the beams of IR light to reflect off objects they encounter intheir path and be captured by the depth sensor, the preference analysismachine may determine the location (e.g., the distance from the depthsensor) of the objects off which the beams of IR light reflected (e.g.,the user, a furniture piece, or a wall). In various example embodiments,based on the received spatial data, the system may determine details ofthe objects in the room, such as spatial measurements of the objects inthe room (e.g., the dimensions of the user's body). The cameradetermines one or more measurements (e.g., dimensions) of the body ofthe user as part of the analysis of the image and the model. Theprocessor with 3D model information from the user may also determine,based on the measurements of the user's body, one or more sizes offashion items from different brands (e.g., manufacturers or sellers offashion items) that may fit the user's body.

One embodiment for clothing includes obtaining a 3D model of a userstanding in front of the display and rendering one or more articles orproducts on the 3D model. Images such as photographs or videos can bemade of the user (also referred to as customer or client) when trying ondifferent articles. These images can be seen on the display and cansimply be ordered/edited/deleted by the user by “dragging” them acrossthe screen. In this example, the display is designed as a touch screen.In this manner, the articles tried on can be compared more realisticallyand more easily by the user.

Footprint for Authentication or Identification

FIG. 1D shows an exemplary system for identifying people based on footprints. The top of FIG. 1D shows exemplary left and right foot printregions that are analyzed to uniquely determine a person forauthentication, ecommerce as part of a multi-factor authentication, babyidentification, or criminology purposes, among others. FIG. 1D nextshows exemplary hallucal area patterns near the big toe, while thebottom of FIG. 1D shows ridge counts and shape at the core of thepatterns. The numbers in each case indicate the count for each pattern.On the ball of the foot there are five pattern bearing areas. One isproximal to the great toe (the hallucal); three others lie lateral tothe hallucal, below the small toes, and together form the plantar area;and the fifth, or hypothenar, is located on the lateral edge of the soleproximal to the third plantar area. The first plantar area lies nearestthe hallucal. Here we find the open field (0), the upright loop openingdistally (U), the inverted loop opening proximally (U), and the whorl(W). The hypothenar area contains principally loops (U), rarely a whorlor an arch. These five areas may be likened to the five finger tips.

CT Scan of Shoes

While shoe manufacturers often have the 3D dimensions of their productsfor user fitting purposes, some may not. A high speed shoe scanner isused in such cases. As shown in FIG. 1E, a CT scanner can provide insidedimensions of the footwear that can be used to best-match the user'sfoot. Additionally or instead, the system may receive measurement datafor at least some footwear models via user input, via a communicationfrom manufacturer of the footwear, or both.

As shown in FIG. 1E, the X-CT scan system according to the embodiment ofthe invention comprises a main controlling and data processing computer33, a base 34, a shoe rotary support and mechanical control devicethereof 32 placed at the center of the base 34 and for placing the shoeto be modeled in 3D, a X-ray generation device 31 and a data acquirementsystem which are at two sides of the base 34. The data acquirementsystem comprises a planar array detector 35, a readout circuit andcontrol logic unit for detecting X-ray projection data and theprojection data on the detector. The main controlling and dataprocessing computer 33 is responsible for the main controlling duringthe X-CT system operation, and processes the project data obtained bythe data acquirement system to reconstruct a three-dimension image ofthe whole the shoe, and display it on a display. One side of the planararray detector 35 is leveled to the prolong line of the connecting linebetween the X-ray source 36 of the X-ray generation device 1 and thecenter of the shoe rotary support. That is to say, the planar arraydetector 35 is axially deviated with one side thereof passing the axis.The X-ray source can be an X-ray tube, an accelerator radiation sourceor an isotope source, which depends on the shoe size and applicationcircumstance. The number of the detectors in the data acquirement systemis reduced by half than that in the convention one, and the projectiondata related to the entire acquirement system is reduced by half.

The control, data transmission and image reconstruction of the CT systemis executed by a computer workstation. The scan control information,position information and projection data are transmitted through thedata acquirement system to the computer workstation, which performs thethree-dimension image of the shoe, and displays it in three-dimension ona display. In order to precisely reconstructing the image, the X-rayimaging system should accurately measure or define the followingparameters, the distance D from the X-ray source point to the detector,the distance R from the X-ray source point to the axis of the rotarysupport, the mapping position P(θ, u, v) on the imaging screen of theX-ray source point, the pixel size dx of the imaging screen, and therotary degree θ of the rotary support. According an embodiment, thereconstruction algorithm uses the cone-beam rebinning method. Firstly,the cone-beam projection data intercepted in 360 degree scope derivedfrom the output of the detector is rebinned to parallel-beam projectiondata of 180 degree scan scope. A complete three-dimension image of theshoe is reconstructed through a convolution back-projection method.

Depending on the information available, some additional measuredparameters may be assigned a numerical or descriptive value representingthe measurement. For example, one particular model of running shoe mayhave approximately 1 cm of stretch in the heel area. Another model mayhave a high level of cushioning. In the data set, for this shoe modelthe measurement parameter for cushioning may be set to “high” or asimilar numerical value representing a high level of cushioning.Alternatively, some additional measurement parameters may be assignedmerely a binary value representing a true/false or yes/no value,indicating whether or not the footwear model exhibits this additionalparameter. For example, a running shoe having racing spikes may onlyhave an associated value of “yes” (or “true” or “1”) as the value for a“racing spikes” parameter in the data store.

In addition, the parameters may include additional retail-specificparameters. For example, information related to consumer ratings can bestored in the data set. Similarly, information such as return orreplacement numbers and reasons for return can be stored in the data setto provide additional information related to a particular wearable item.

The computing device can access the data set to retrieve the storedinformation related to the user-selected item, and analyze the storedmeasurements and parameters associated with the user-selected item. Thecomputing device may also prompt the user to provide sizing information.For example, the system prompts the user to provide the size that he orshe typically wears in a running shoe. Alternatively, the system mayretrieve the user's primary size from information previously provided bythe user, such as a user profile or previous purchase data. After thesystem receives the user's primary sizing information, the system maydetermine a recommended size of the user-selected item and provide therecommendation. The determined size may be a primary size or analternate size, depending on the model selected and whether or not theselected size of that model runs true to fit. The size determinationprocess will be described in more detail below.

Insole Sensors and Environmental Controller

FIG. 1F shows the insole with foot sensors for the above data capture.The insole can optionally include heater/cooler for comfort. The coolercan run during the summer months, while the heater units operate in thewinter months. Based on a remote setting, the heated/cool Insoles heatto a specific temperature, providing the user with just the right amountof warmth. Like the thermostat in the home, heated Insoles reach thetemperature of the user-then temporarily turn off—coming back on whenmore heat or cool is needed. These insoles are designed to warm up tospecific temperatures where you will remain warm and comfortable, butdon't overheat and sweat. An insulated layer is provided between thefeet and the bottom of the shoes or boots to stop heat/cool air fromescaping. The water-resistant fabric liner absorbs any moisture, and themolded, flexible polyurethane material provides the dry comfort.Removable, replaceable batteries can b used to swap them out withouthaving to take the insole out of the shoes or boots. A esilient,shock-absorbing Poron battery cover is used for added cushion andprotection. The device of FIG. 1F may also be a sole of a shoe, asandal, a sock, sock type device, or a boot. The device contains aplurality of sensors connected by connections to a control electronic.Sensors may be resistive pressure sensors, but are not limited toresistive pressure sensors and can be a variety of other types ofpressure sensors as well as other physiological and biomechanicalsensors. In either case, the controller receives raw data from pressuresensors and generates pressure data for transmission to a local basestation or a smart phone. As many sensors as needed are contemplated. Atransmitter is used to transmit the pressure data to the base stationsuch as a Bluetooth PC or mobile phone, for example. In operation,pressure sensors sense pressure from a foot placed on them. The pressuresensor can be piezoelectric sensors, capacitive sensors, or resistivesensors. For example, where pressure sensors are resistive sensors, theresistance in primary pressure sensors varies as different pressureand/or force is applied to them. The controller sends current throughpressure sensors and determines the pressure at each pressure sensorfrom the resistance detected. Based on the pressure, the shoe can becustomized to compensate for any unsuitable pressure experienced by thewearer and optimize the walking experience.

In addition to pressure sensing, other personal data can be captured.For example, the sensors can include foot bio-impedance sensors that usebioelectrical impedance analysis (BIA) to estimate the heart rate byamplifying the pulsatile impedance component superimposed on the basalimpedance. One embodiment detects the heart rate (HR) from bioimpedancemeasured in a single foot. Four electrodes are used for measurement ofbioimpedance signal; two electrodes for injecting current and the othertwo to capture the voltage signal from human body. The bio-impedancesignal shows deflections corresponding to systole and diastole activityas a measure of heart rate. The electrodes embedded in the footwear canapply a 50 kHz voltage between the outer electrode pairs and measure thedrop in voltage across the inner electrode pairs in one embodiment. Animpedance converter AD5933 separates impedance into real and imaginarypart using discrete Fourier transform. The real and imaginary values ofthe measured bio-impedance signal are processed by a processor to obtaina continuous signal. The bioimpedance signal obtained after de-noisingusing adaptive thresholding. For heart rate detection, synchronousdemodulator plays vital role by demodulating the bio-impedance signalfrom current carrier. To achieving high CMRR in signal in analogdifferential synchronous demodulator for AC signals, the signal issynchronously demodulated using the floating-capacitor with high CMRR.An impedance analyzer is used for getting bio-impedance signal. Waveletthresholding methods can be used for noise removal where waveletcoefficients are threshold in order to remove their noisy part.

Another embodiment measures heart rate and/or EKG with sensors directlyprovided in the footwear or using external wearable devices and suchdata combined with foot-ground contact information is used forambulatory estimates of maximal aerobic power from foot. The user'smaximal rate of oxygen uptake sets the upper limit for sustainedphysical activity and is the standard measure of aerobic fitness.

The footwear can include piezoelectric elements that generateelectricity and also can be actuated to cancel or dampen vibrations. Inthis piezoelectric embodiment, a shoe body is positioned above a basethat includes a plurality of spring blades or leaf spring elements. Eachspring blade has a top portion that extends from the base in a curvedmanner and a bottom portion secured to a blade foot. The blade can beplastic springs, which get compressed with each foot strike and recoilas the wearer proceeds through the gait cycle. In one embodiment, thebase and the spring blade can be a piezoelectric composite that isdirectly molded into the shape of the base and the spring blades. Thebase can be an energy storage device, as detailed below. In anotherembodiment, the base is connected to individually tuned blades designedto propel runners. In one embodiment, 16 blades are composed of a highlyelastic, piezoelectric polymer that is angled forward for high energyreturn in any environment. In addition, each blade is tuned to providesupport in each phase of a runner's stride. The shoe can spring backwith each step and propel a runner forward. The blades have differentthicknesses and angles, which can influence the amount and direction ofthat energy return. Each blade is “tuned” differently to correspond toits position on the shoe, as well as to take into account body mass; menand women will get shoes with the appropriate amount of flex andresponse.

In an embodiment, the base can have a flexible and stretchable batterycomposed of strain free LiFePO4 cathode, Li4Ti5O12 anode and a solidpoly ethylene oxide (PEO) electrolyte as a separator layer. Featuringsolid thermoplastic electrolyte as a key enabling element this batteryis potentially extrudable or drawable into fibers or thin stripes whichare directly compatible with the weaving process used in smart textilefabrication. In an embodiment, a materials system for the design of adrawable lithium polymer battery with a view of eventually obtaining abattery-on-fiber is disclosed. The cathode material used in oneembodiment is LiFePO4. While the base or sole can be the battery, theupper or flexible portion covering the foot can also include the batteryor a suitable energy harvesting device such as solar cell.

In one embodiment where the insole is strengthened at the bottom tobecome a sole, the shoe has an upper surface (upper) that can be fabric,leather, or synthetic materials for cooling/warming the foot. Oneembodiment of the shoe covers the upper surface with a solar electricitygenerator whose output is connected to a power regulator. Inside theshoe, a sole contains a piezoelectric device covering the entire foot isused to generate electricity. In one embodiment, the sole can be a piezosole from Smart-Material Corp. of Sarasota, Fla. The shoe also containselectronics such as CPU, transceivers, energy store (such as batteriesor super-capacitors) and power regulator. The regulator can acceptelectric output from a variety of sources including piezoelectricdevices and solar cells, for example. The sole plate can include aconnector such as a micro USB connector to recharge a flexible energystorage device such as a flexible battery, or alternatively can be aflexible supercapacitor. The shoe can also receive recharge throughinductive charging or other wireless charging systems. In oneembodiment, the human foot electrically touches a pad on the shoe toprovide contacts for a BAN communication network with sensors mounted onthe wearer's body. As noted above, the sensors can be an EMG detector,EEG detector, an EKG detector, an ECG detector, a bioimpedance sensor,an electromagnetic detector, an ultrasonic detector, an opticaldetector, a differential amplifier, an accelerometer, a video camera, asound transducer, or a digital stethoscope. The bioimpedance sensor candetermine one of: total body water, compartmentalization of body fluids,cardiac monitoring, blood flow, skinfold thickness, dehydration, bloodloss, wound monitoring, ulcer detection, deep vein thrombosis,hypovolemia, hemorrhage, blood loss, heart attack, stroke attack. Thesensor can also communicate with an indoor position sensor, a motionsensor, a door sensor, a bathroom sensor, a water overflow sensor, anexercise equipment sensor, a smoke detector, an oven sensor, a cookingrange sensor, a dish washer sensor, a cabinet door sensor, arefrigerator sensor, a refrigerator container sensor, a kitchen waterflow sensor, a dish sensor, a bowl sensor, a chair sitting sensor, asofa sitting sensor, a bed sensor, a weight sensor, a television viewingsensor, a radio listening sensor.

Insole Weight Scale

The shoe can contain a weighing system to provide real-time measurementof the wearer's weight. Alternatively, a shoe pad insert or a sock or anadhesive band-aid can contain the weighing elements for temporarilymeasuring the wearer's weight. The system can have an array of forcesensing resistors or weight sensitive sensors such as piezoelectricsensors. One embodiment can provide a resistor array in the form of aresistive (carbon) sheet with interdigitated contacts that can beshorted when weight is applied to change the overall resistance of theresistor. The resistor is calibrated to convert a predeterminedresistance to a particular weight. The resistor can be flexible polymersto deal with loading conditions. In a liquid sensing embodiment, a fluidcavity or an array of fluid cavities is provided with a liquid that isdisplaced or pressurized as a function of weight. The array of cavitiesmay be constructed to handle different pressures and sensitiveness. Forexample, a large cavity can be positioned under the foot section thatabsorbs most of the weight, and a small cavity can be positioned nearthe center of the foot as that part receives light pressure. With thesingle or array of cavities, a calibrated pressure sensor is used todetect weight. Alternatively, a pressure sensor can be embedded in theshoe, insert, sock, or band-aid to directly measure weight without thefluid cavity. The system can include temperature and altimeter sensorsto better predict weight and to capture health parameters, for example.Using the sensors, a wearer can review his or her weight at nearly anytime. Runners using such a system and device to know their hydrationloss; chiropodists may wish to monitor weight distribution over apatient's feet; and athletic trainers may wish to analyze weightdistribution and forces. In one embodiment, only a portion of the footneed to be covered, covering a certain percentage of the overall weight;and that percentage is scaled to a user's full weight. Weight andcompression forces monitored in a shoe or shoe insert, in accord withthe system, can further assist in gauging caloric and/or physicaleffort.

The weight sensors can communicate with processor in the sole or cancommunicate with a wireless phone using a personal area network such asBluetooth or with a remote processor using WiFi, for example. The shoeweight is known and can be subtracted from the total weight to arrive atthe wearer weight. Precise weight and heart rate measurement data can beused as part of a population health management system to keep patientweight to an ideal health. The weight information can be used to detectshort term loss of water such as after marathon, race, soccer/footballgame, or intense outdoor activities such as hiking, for example. Ifweight loss indicative of dehydration, the processor can let the wearerknow to drink water and rehydrate, for example. As the sensors alsodetect foot/ground impact, they can detect improper walking/runningpostures and report to users or doctors for corrective actions.

A shoe-mounted or shoe-integrated device can monitor shoe usage andindicate when the shoe has exceeded its useful life. The sensor candetect events of the footgear due to activity of a wearer. The eventscan be impact events or rotational events, but are not limited to suchevents. For detecting impact events, the sensor can be an accelerometerresponsive to motion. For detecting rotational events, the sensor can bea Hall-effect sensor, which can be responsive to a rotating element,such as a Ferris element. The processor can count the events detected bythe sensor and maintain a cumulative event total. The processor can thencompares the cumulative event total to an event threshold, which can becalculated from the predetermined number of events and an individualizedfactor. The individualized factor can include at least one of thewearer's weight, the climate where the footgear is worn, a type ofpredominate surface on which the footgear is worn, the wearer's age, thewearer's foot pronation or running style (such as whether the user is aheel striker or toe striker), and the wearer's injury history. Thesensor, processor, and display can be secured within the footwear andcommunicates with a smart phone or a computer using a PAN such asBluetooth and the sensor can be powered by a coin cell battery, solarcell, or piezoelectric electric generator.

The device can estimate distances run or can measure the shoe'soperating parameters, such as cushioning. In a particular embodiment,the device can include a sensing unit, a programmable processorinterpreting data from the sensing unit, and an indicator for notifyingthe wearer of the shoes' status. The processor can be programmed duringmanufacturing to incorporate typical variable values that are relevantto measuring shoe life. The processor can also be field programmed bythe retailer or end user to enter individualized, wearer-specificvariable values.

In one embodiment, a wear monitor can indicate when a shoe or componentmay have exceeded its expected useful life. The indication can betriggered by a measure of use, such as steps taken or distance accruedin the shoes, either through estimation or actual measurements. Themonitor can take into account varies parameters related to the wearer ofthe shoe and environmental factors to more accurately determine when apair of shoes has reached a wear out period. By employing sensors, themonitor can also be measure certain operating parameters of the shoe,such as the loss of a critical amount of resilience, and indicating tothe wearer that the shoes are no longer adequate to protect the wearerfrom injury. The wear monitor can be fabricated into the shoe duringmanufacturing or can be a portable stand-alone device and can employvarious technologies to provide a status indication to the wearer.

Another particular embodiment can include a method for estimating wearto a component of footgear based on the expected functional life of thecomponent, as determined by a predetermined number of events. The methodcan include calculating an event threshold based on the predeterminednumber of events and an individualized factor, counting the eventsdetected by a sensor, maintaining a cumulative event total, andcomparing the cumulative event total to the event threshold. The methodcan also include displaying a representation of the comparison on adisplay device.

The individualized factor can include at least one of the wearer'sweight, the climate where the footgear is worn, and a type ofpredominate surface on which the footgear is worn, the wearer's age, thewearer's foot pronation or running style (such as whether the user is aheel striker or toe striker), and the wearer's injury history.

The processor can calculate an event threshold based on thepredetermined number of events and individualized factor, count theevents detected by a sensor, from the counting, maintain a cumulativeevent total, and compare the cumulative event total to the eventthreshold. The individualized factor can include at least one of thewearer's weight, the climate where the shoe is worn, and a type ofpredominate surface on which the shoe is worn, the wearer's age, thewearer's foot pronation or running style (such as whether the user is aheel striker or toe striker), and the user's injury history.

Mass-Customized Sole/Insole Design Process

The above processes can arrive at the inside dimensions of a particularshoe that best fits the user. However, the standard shoes may still needfurther customization to support the user's particular body dimensionssuch as arches. Thus, a mass-customized insole is detailed next.

One embodiment is a diagnosis and a system for design ofpatient-specific orthotics focused principally on dealing with thekinetics of pronation. In the functioning foot there are specificrelationships between the anatomical structures commonly identified fromboth the frontal plane and the sagittal plane of reference. Instabilitycan result from a misalignment between the forefoot and rear-foot whichprevents the foot from functioning in a fully integrated manner. Howeversuch a simple structural (kinematic) classification as this overlooksthe critical matter of how muscular energy is transmitted throughanatomical structures in such a way as to confer normal motion (kineticfunction) on the foot. For example, the pronation force about thesub-talor joint axis is known to increase as a result of structuralmisalignment. But an analysis in kinetic terms would account for theorigin and magnitude of the pronation force and why this force affectsthe sub-talor joint. Once the problem is presented in kinetic terms, theanatomical structures are seen to play their part in the resolution andtransmission of forces rather than suggesting their source.

The system also models kinetic processes in the foot using Kirby'sdynamic equilibrium between the sum of pronation and supination forcesoccurring about the sub-talar joint axis. (“Rotational Equilibrium”theory (Kirby, K. A. 2001 “Sub-talar joint axis location and rotationalequilibrium theory of foot function” JAPMA 91(9): 465-487)), the contentof which is incorporated by reference. Assessed from the sagittal planeof reference, the foot has been described as a compound pivot made up ofthree key pivots. The three key sagittal plane pivots can be named the“Heel rocker” the “Ankle Rocker” and the “Forefoot Rocker”. Footpronation results when a restriction occurs at either the ankle pivot orthe forefoot pivot during gait. Restriction is revealed by the inabilityof the ankle or forefoot rocker to function normally. Restriction can beanatomical or physiological in origin and its extent can be influencedby footwear or orthotics or both. If restriction at a key pivot sitespersists of foot becomes chronically unstable, pronation becomesendemic. This process can lead to deterioration in pivotal function andfurther instability.

The fabrication of an orthotic insert or shoe for a patient's foot caninclude providing shoe sensors with pressure sensors or accelerometers(as detailed in FIG. 1F which shows exemplary footwear with sensors andheater/cooler embedded therein) to the user to pace the foot to one ormore of the following tests and ascribing a test value(s) within apredetermined set of relative values for each test which is indicativeof one or more properties of the patient's foot:

(i) supination resistance test (as defined); and

(ii) Jack's test (as defined);

(b) recording each test value in a database;

(c) comparing the test values to control values indicative of one ormore predetermined orthotic designs stored in the database; and

(d) selecting an orthotic design(s) from the predetermined orthoticdesigns dependent on that comparison.

The process may further include one or more of a skeletal integritytest, a fascial chord tension test, an ankle joint stiffness-lunge test,a principal activity velocity test, a sagittal plane morphology test,and a hamstring stiffness test. More explanation of the various tests isas follows:

(a) Supination Resistance Test—This is the amount of force required toresupinate the foot. With the patient standing in a relaxed weightbearing position, the force is graded on various levels and recordedfrom very low to very high. This index reveals where the centre ofpressure is to be applied to the foot by the orthotic device, whethertowards the back or the front. Foot integrity is also estimated from theamount of change in arch amplitude observed when the foot goes from anon-weight bearing position to a weight bearing. The change in archamplitude may be measured within a range of five increments categorizedfrom very low to very high; if the amplitude changes by two increments,the foot is classified as a foot with poor integrity, whereas if thechange is just one increment the foot would be classified as one withgood integrity. If there is no change then the integrity measure isscored as excellent. These integrity measures give further informationfor application of the design parameters that relates to the amount ofrear foot to fore foot support.

(b) Windlass mechanism test—Jack's Test and Fascial Chord Tension Test.The force required to lift the hallux when the patient's foot is in afull weight bearing position is determined by The Jacks Test. When thehallux is lifted, the foot automatically begins to resupinate. The forceto initiate the foot resupination is graded on three levels form low tohigh. This index provides additional information as to the placement ofthe centre pressure in the orthotic design. Fascial Chord Tension Testis as follows. With the foot non-weight bearing, the first metatarsal isdorsi-flexed and the prominence of the fascial chord is recorded. Theprominence of the fascial chord is graded from low to high. Thisparameter is important as this allows the design to be modified toaccommodate the fascial chord by way of a fascial groove. It isimportant to be able to adjust the design this way to help protect andfacilitate the windlass effect. The orthotic design may require furtheradjustment including wedging in the rear foot to help push the chord outof the way.

(c) Sagittal plane morphology test. This categorizes the foot in termsof the gradient, the anterior calcaneal surface and the foot apexposition. The gradient is evaluated as low, medium, or high. The footapex position when combined with the gradient is categorized as rear,central, or forward, providing key information on the amount of softtissue that surrounds the anterior heel area and can affect the amountof rear foot orthotic contour applied in the design.

(d) Hamstrings tension test. This is a test indicating the amount oftension in the hamstrings so as to determine the possible compensatoryimpact on the ankle joint in the close kinetic chain. Hamstring tensionis graded on three levels low, medium and high. When the tension iscategorized as high changes are made to the design so as to facilitatesagittal plane function.

(e) Lunge test. Failure in this test implies that greater ankle jointfacilitation must be provided for in the design. The design will reflectthe increased force needed to establish foot resupination.

(f) Principal activity velocity test. The principle activity velocity isdefined as the level of activity the device is being designed forwhether that is predominantly standing or moderate walking or running.The activity is graded on three levels from low to high. This isrecorded as an index. When applied to the design it influences whetherthere is a need to more closely contour to the foot type or wedge morethe rear foot area of the orthotic. The greater the velocity the greaterthe force of correction required and the further back the device apexshould be.

One exemplary process applies the following Criteria for Classificationof Foot Posture (Cross and Lehman)

Supination of the Subtalar Joint

-   -   Inverted calcaneus    -   Shallow concavity superior to the lateral malleolus    -   No concavity inferior to the lateral malleolus    -   High arch

Neutral Subtalar Joint

-   -   Vertical or slightly inverted calcaneus    -   Even concavities above and below the malleoli    -   No curvature of the Tendo Achilles (TA)    -   No bulging of the medial aspect of the foot    -   Discernable medial arch

Pronation of the Subtalar Joint

-   -   Eversion of the calcaneus (relative to the lower leg), but only        to vertical (relative to the weight bearing surface)    -   Bulging of the medial aspect of the foot    -   Long sloping concavity superior to the lateral malleolus    -   Small deep concavity inferior to the lateral malleolus    -   Flattened medial arch (differentiated from a pathological pes        planus on external rotation of the weight bearing leg: if the        arch configuration changes with external rotation of the lower        leg the indication is that the arch can recover).

Hyperpronation of the Subtalar Joint

-   -   Eversion of the calcaneus (relative to the lower leg), beyond        vertical (relative to the weight bearing surface)    -   Bulging of the medial aspect of the foot    -   Long sloping concavity superior to the lateral malleolus    -   Small deep concavity inferior to the lateral malleolus    -   Flattened medial arch

Helbing's sign may also be apparent

The design process begins in several different ways. One option is toload a scanned foot model. Another option is to load a generic, modifiedinsole template. Generic templates of various sizes provide a usefulstarting point for the design of custom insoles. Another option is toload a third-party patient data file, which may contain informationabout pressure data (dynamic forces and pressure distributions createdin the patient's foot while walking), laser scan data, or medical DICOMfiles as mentioned above. Together, the insole file, image file, anddata files may be saved into a single patient insole file. An insole CADsystem can be used to edit the foot model. The edit functions generallyallow insole design personnel to add protrusions or carve out recessesin the insole to accommodate user specific requirements. For example,the PAD and MTT (Metatarsal) functions create pads on the surface of theinsole that serve to redistribute forces in the patient's foot.Similarly, the CIRCLE and POCKET functions create recesses to alleviatepressure on injured or irregular surfaces of the foot. In addition tothe four functions described, the EDIT menu in the Insole Modelerincludes the following edit functions: Height Front, Height Back,Measuring, MultiPoint, Area, Plateau, and Arch Support. The HEIGHT FRONTand HEIGHT BACK functions are designed for elevating parts of theinsole, which may be used for eliminating surplus elements on the frontor the back of the insole, thereby making the insole thinner or forcreating shoe fillings in cases of amputated feet or other deformities.The MEASURING function calculates distances between points in the modeland can preferably provide linear as well as coordinate distances inpixels and inches or millimeters. The MULTIPOINT function is a true 3Dfunction for generating new surfaces defined by multiple pointsinterconnected by lines. This function is useful for deepening orraising the edges of the insole, for creating channels for releasingpressure from the plantar fascia, or for designing a heel cup, which isimportant in cases of tendonitis, bursitis, and partial or totalruptures of the Achilles tendon. The AREA and PLATEAU functions aresimilar in that they are free form raised or recessed areas definedpolygonally by setting points on the surface of the insole. Thedifference between the two is that in the AREA function, the recess orraised portion converges to a point whereas the plateau rises or fallsto a flat surface. The ARCH SUPPORT function is one of the most commonlyused functions in the Insole Modeler 930. The function is relativelyself-explanatory and is used to add outer support for the longitudinalarch area. The Final Adjustments Functions 1050 generally permitlarge-scale modifications to the insole. For example, the thickness ofthe insole is modified by the LIFT UP or LOWER DOWN functions while thelateral tilt is altered using the PRONATION or SUPINATION functions. Theheel of the insole is defined by specifying the HEEL LENGTH, CROSSINGLENGTH and HEEL DELTA HEIGHT parameters. The Global Change functionsallow modification to the insole as a whole. For example, the SMOOTHINGfunction is used to eliminate uneven surfaces created during thescanning procedure or to smooth sharp edges created by local editingfunctions. The SCALING function allows the designer to change the scaleof the insole along any or each of the three Cartesian coordinates (i.e., X, Y, or Z axes). The ZOOMING function permits insole modelers toview the insole from different perspectives and with differentmagnifications. The MIRRORING function permits the copying of existingfeatures about a user-defined mirror axis. The final insole 3D model isthen sent to a mass-customized insole 3D printing system as detailedbelow.

3D Printed Soles

FIG. 1G shows an exemplary template for rapid, inexpensive 3D printingof insoles using engineered sand technology as described in more detailsbelow. In this system, a plurality of templates is provided, one foreach foot size such as 7, 8, 9, 10, 11 and in-between sizes. The rightsize for the user is selected and inserted into the bed of sand and acomputer controls actuators 41A-41C to customize the curves to theuser's dimensions. Sand is flowed over the customized templates and set.The template is removed, and rubber or suitable insole material ispoured into the sand bed and cured and removed for finalcleaning/polishing if needed. The sand bed can be reactivated foranother insole or sole. For forming custom shoes, the insole can simplybe padded with more materials, and an outer can be stitched or glued tothe sole.

Each template is mechanically connected to an array of linear actuatorsthat can morph the dimensions of the insole to personalize the insole tothe specifics of the user. In the embodiment of FIG. 1E, a number ofspecific spots are adjusted such as the deep heel cup region, theorthotic arch support region, the foot bed region, and the forefootregion. The linear actuators 41A, 41B, and 41C can be electric orhydraulic or shape actuated materials such as nitinol and can push orpull the insole to provide the desired cusps to best fit the user.

While conventional additive manufacturing 3D printers can be used formass-customization of the shoes, the material available is limited andthe print speed is slow, leading to fragile and expensive shoes. FIGS.1H-1I show exemplary systems and techniques for manufacturing in volumewith a wide range of materials are disclosed for fabricating shoes atmass customization scale. The system can also be further cleaned upafter manufacturing using CNC for smoothing the soles, stiching fabricsonto the sole, or any other required post-processing manipulation of thefabricated shoes.

In one aspect, systems and methods are disclosed for shaping areformable material by holding a volume of particles inside a containerhaving a first elastomeric membrane surface; infusing the volume with aliquid to mobilize the volume of particles; and pressing a master shapeinto the membrane with atmospheric pressure.

In another aspect, a method to form an object includes infusing a liquidinto a container having a first elastomeric membrane surface; pressing amaster shape into the membrane with atmospheric pressure; and shaping areformable material into the object according to the master shape.

In yet another aspect, a method to form an object includes infusing aliquid into a container having a frame with first and second elastomericmembranes; a first port to deaerate the volume of particles; and asecond port to infuse the volume with a liquid for mobilizing the volumeof particles; pressing a master shape into the membrane with atmosphericpressure; and shaping a reformable material into the object according tothe master shape.

Implementations of the above aspects may include one or more of thefollowing. The volume of particles can be deaerated. The liquid can beextracted through one or more screen elements placed proximal to thevolume of particles. The atmospheric pressure continues to hold theparticles in place against the elastomeric membrane when the mastershape is removed from the outer surface of the membrane. The methodincludes heating and driving liquid from the particle volume. A residueof a binding adhesive is left to lock the particles into a continuousforce-resisting mass. A complementary shape is impressed to the mastershape in the membrane. A rigid outside frame can be used with top andbottom elastomeric membranes facing the top and bottom surfaces of thecontainer. The master shape can be pressed against the top elastomericmembrane of the container by atmospheric pressure. The pressingoperation includes applying a flexible vacuum cap which is sealed overthe shape and against the container's top surface membrane; evacuatingair from a space between the top membrane and the vacuum cap; extractingliquid from the volume; and pressing the particles within the containerby atmospheric force acting in opposed directions against the vacuum capand the bottom surface membrane. Air can be introduced into the vacuumcap, and then the cap and the master shape can be removed from theformed surface of the elastomeric membrane. The container is formedagainst the master shape. The method includes placing the master shapeon an air-impermeable surface; placing a membrane of the container overthe shape; and placing a vacuum cap or a vacuum-bagging film over thecontainer to effect forming of the elastomeric membrane against themaster shape. An envelope with a vacuum seal on its perimeter can beused to contain a mass of particles and to extract air from between themaster shape and the envelope. The master shape can be placed on the topelastomeric surface of a first rigid-framed container and a membranesurface of a second container can be placed over the master shape. Thesecond container fits inside the frame of the first container and avacuum cap is placed over and sealed outside the second containeragainst the surface membrane of the first container. The method includesevacuating the volume under the vacuum cap and pressing the master shapebetween the elastomeric sides of the first and second containers. Theliquid is extracted so that the two volumes of particles are pressedtogether and against the membranes surrounding the contained shape. Thevacuum cap can be vented with air and removed; the top container canthen be removed; and the shape can then be removed from the membrane ofthe bottom container. The top container can be placed over the bottomcontainer; and forming a closed, shaped cavity complementary to thesurface of the master shape used to form the cavity. Two identicalcontainers of either the first or the second container can be pressedaround a master shape with or without using the vacuum cap. Thecontainers can be joined and sealed by either a seal mounted on one orboth of the containers or by seals mounted on a seal ring which fitsbetween the two containers. The liquid can be extracted prior to themaster shape being removed from the shaped reformable material. Theliquid can be withdrawn to leave a residue of liquid on the shapedreformable material; and solidifying the residue. The method can includepreforming a surface material over the master shape as withthermoforming or additive processing. The container walls can be air andliquid impermeable. An inelastic formable surface can be used thatconforms to the master shape surface. A surface can be formed over themaster shape to conform to the master shape and the shaped materialsurface can be pressed against the volume of particles without deformingthe shaped material surface. The method includes providing a releasesurface to the master shape; pressing the master shape against thevolume of particles to form the object against the release surface; andremoving the object from the master shape with the release surface. Therelease surface can be applied to the master shape with a surfaceelement covering the reformable material surface not overlaid with themaster shape surface.

In another aspect, an apparatus to form an object in accordance with amaster shape includes a container to hold a volume of particles, saidcontainer having a first elastomeric membrane surface; a first port todeaerate the volume of particles; and a second port to infuse the volumewith a liquid for mobilizing the volume of particles; and a presscoupled to the container to move the master shape into the membrane toshape a reformable material into the object according to the mastershape.

Implementations of the above aspect may include one or more of thefollowing. One or more screen elements can be placed proximal to thevolume of particles to extract the liquid. Atmospheric pressure can beused to hold the volume of particles in place against the elastomericmembrane when the master shape is removed from the membrane. A heatercan be used to heat and drive liquid from the particle volume. Thecontainer can be a rigid outside frame and top and bottom elastomericmembranes facing the top and bottom surfaces of the container, andwherein the master shape is pressed against the top elastomeric membraneof the container by atmospheric pressure. The apparatus can include aflexible vacuum cap sealed over the shape and against the container'stop surface membrane; a third port to evacuate air from a space betweenthe top membrane and the vacuum cap; and pressing of the particleswithin the container by atmospheric force acting in opposed directionsagainst the vacuum cap and the bottom surface membrane. Air can beintroduced into the vacuum cap and then the cap and the master shape canbe removed from a surface of the elastomeric membrane. The master shapecan be placed between an air-impermeable surface and the membrane of thecontainer and wherein a vacuum cap or a vacuum-bagging film is placedover the container to form the elastomeric membrane against the mastershape. An envelope with a vacuum seal on its perimeter can be used tocontain a mass of particles and to extract air from between the mastershape and the envelope. The master shape can be placed on the topelastomeric surface of a first rigid-framed container and placing amembrane surface of a second container over the master shape. The secondcontainer fits inside the frame of the first container and a vacuum capis placed over and sealed outside the second container against thesurface membrane of the first container. A vacuum pump can evacuate thevolume under the vacuum cap and press the master shape between theelastomeric sides of the first and second containers. A pump can extractthe liquid so that the two volumes of particles are pressed together andagainst the membranes surrounding the contained shape. The vacuum capcan be vented with air and removed; the top container is removed; andthe shape is removed from the membrane of the bottom container and thetop container is placed adjacent the bottom container to form a closed,shaped cavity complementary to the surface of the master shape used toform the cavity. The first and second containers can be identical andcan be pressed around a master shape without using the vacuum cap. Thecontainers can be joined and sealed by either a seal mounted on one orboth of the containers or by seals mounted on a seal ring which fitsbetween the two containers. A seal ring can be used to channel vacuum orair pressure between the containers and to hold the master shape in aprecise orientation and position between the two opposed containers. Anexpander can be used within the container to press the particulatematerial against cavity walls of the container. The apparatus caninclude a second container cooperating with the first container to forma complementary cavity from the master shape; and a third containerplaced in the complementary cavity to replicate the master shape. Arigid frame or a flexible-edge frame can be used. The frame can form acontinuous surface complementary to a master shape's surface. A secondelastomeric membrane can be used, and the elastomeric membranes canoverlap or abut each other. Additional containers each having a membranecan be used with the container's membrane to form a continuous surfaceof membranes. Further, additional containers can be used to form a shapecomplementary to the interior of a master cavity.

In another aspect, an apparatus to form an object in accordance with amaster shape includes a container to hold a volume of particles, saidcontainer having a frame with first and second elastomeric membranes; afirst port to deaerate the volume of particles; and a second port toinfuse the volume with a liquid for mobilizing the volume of particles;and a press coupled to the container to move the master shape into themembrane to shape a reformable material into the object according to themaster shape.

Implementations of the above aspect may include one or more of thefollowing. The second membrane is bonded to the frame. The firstmembrane is mounted to a seal. A clamp can secure at least one membraneto the frame. One or more ports can be provided on the frame. Liquid,evacuation, and vacuum-activated seal tubes can be mounted to the frame.A rim evacuation screen element can be positioned in the frame. Theframe can be rigid or flexible. A vacuum activated seal can be providedon the frame. A tube can be used for evacuating and filling thecontainer. Double layer screens having feed elements to distribute andextract liquid through the volume of particles can be used. One or morescreens can be used to conform to the master shape. One or more internalscreens can be mounted with the particles flowing on both sides of eachinternal screen. The frame can have one or more containers joinedtogether around the master shape or alternatively can have one or morecontainers joined by vacuum seals. One or more feed tubes can connect toan interior element inside the membrane. A flexible spine element can beused within an interior cavity of the container. One or morereinforcement fibers can be used, and in certain implementations, thefibers can be distributed in bundles within the volume of particles. Anair pump or source can be used to provide internal pressurization. Avacuum source can provide a vacuum between a cavity in the container andthe container. An air source and a vacuum source can alternatelypressurize and vent the container to distribute the volume of particlestherein. A seal ring can be used. The seal rings can be mounted againstseals or can be mounted with attached seals. The attached seals can bevacuum activated. A second container can be joined with the containerand wherein a vacuum is formed in an interior of the joined containers.The master shape can be mounted on the seal ring. Flanges can be mountedto control a mating line between opposed membranes of containers. Asecond container can be positioned within a cavity formed by an outsidecontainer. A vacuum seal can be used with a vacuum cap. A vacuum tubecan be used that penetrates through the membrane. A vacuum cap withmounted container can be used in place of the membrane. One or morescreen elements can be placed proximal to the volume of particles toextract the liquid. Atmospheric pressure holds the volume of particlesin place against the elastomeric membrane when the master shape isremoved from the membrane. A heater can be used to heat and drive liquidfrom the particle volume. The container can have a rigid outside frameand top and bottom elastomeric membranes facing the top and bottomsurfaces of the container, and wherein the master shape is pressedagainst the top elastomeric membrane of the container by atmosphericpressure. An envelope with a vacuum seal on its perimeter can containthe mass of particles and extract air from between the master shape andthe envelope. The master shape can be placed on the top elastomericsurface of a first rigid-framed container and a membrane surface of asecond container placed over the master shape. An expander within thecontainer can be used to press the particulate material against mastershapes and against cavity walls of other containers. The apparatus canhave a second container cooperating with the first container to form acomplementary cavity from the master shape; and a third container placedin the complementary cavity to replicate the master shape. A secondelastomeric membrane can be used that either overlaps or abuts theadjacent membrane. Additional containers each having a membrane coupledto the container can be used to form a continuous surface of membranes.Additionally, one or more additional containers can form a shapecomplementary to the interior of a master cavity.

In yet another aspect, a base station is disclosed to form an object inaccordance with a master shape. The base station includes a liquidreceiver; a vacuum source to evacuate air from the liquid receiver; anair compressor, pump or source to generate pressurized air; and acontroller coupled to the liquid receiver, the vacuum source and the aircompressor to form the object.

Implementations of the base station can include one or more of thefollowing. Tubes can be used to provide vacuum and to control the flowof liquids to and from the receiver. Valves, sensors, and other circuitscan be interfaced with the controller. An electrical power source can beused to provide power to operate valves, sensors, the vacuum pump andthe air compressor. The controller can be a menu-driven processcontroller. A heater can be used to vaporize and expel liquid fromcontainers of reformable material. The reformable material createscontours of the master shape or alternatively can be molded against acomplementary surface of an elastomeric membrane. The liquid contains asoluble binder, which can be left on a shaped volume of particles. Thebinder locks a shaped volume of particles in place after the liquid isremoved. The heater can be a radiant heater, a convective air heater,microwave heater, radio-frequency heater, or inductive heater. Theheater can include one or more heating elements within the container.The heater is controlled by the controller. A container can be used tohold a volume of particles, said container having a frame with first andsecond elastomeric membranes; a first port to deaerate the volume ofparticles; and a second port to infuse the volume with a liquid formobilizing the volume of particles; and a press coupled to the containerto move the master shape into the membrane to shape a reformablematerial into the object according to the master shape. Alternatively,the container can have a first elastomeric membrane surface; a firstport to deaerate the volume of particles; and a second port to infusethe volume with a liquid for mobilizing the volume of particles; and apress coupled to the container to move the master shape into themembrane to shape a reformable material into the object according to themaster shape. The container can include a rigid outside frame and topand bottom elastomeric membranes facing the top and bottom surfaces ofthe container, and wherein the master shape is pressed against the topelastomeric membrane of the container by atmospheric pressure. The basestation can also include a flexible vacuum cap sealed over the shape andagainst the container's top surface membrane; a third port to evacuateair from a space between the top membrane and the vacuum cap; andpressing of the particles within the container by atmospheric forceacting in opposed directions against the vacuum cap and the bottomsurface membrane. The master shape can be placed between anair-impermeable surface and the membrane of the container and a vacuumcap or a vacuum-bagging film can be placed over the container to formthe elastomeric membrane against the master shape. The vacuum pump canbe a mechanical pump or an air driven pump such as a Venturi pump. Asecond vacuum pump can be used. Isolating valves can be used, and aregulator and one or more valves can be used to pressurize a liquidtank. A vent valve can also be used to cycle from a vacuum source to apressure source. A three-way valve can route air and vacuum to theliquid tank. A filter can be used to prevent particulate carryover. Anair-liquid separator and/or a level indicator can also be used. Avacuum, pressure, liquid and temperature sensor can provide data to thecontroller for process control. A heat exchanger can be used to condensevapor. A slurry transfer tank can be connected to the container. Thecontainer can be a single unit, or can have a plurality of containersadjacent to or inside the container to form a cavity. The containers canbe mated with a seal ring.

In yet another aspect, a method to shape a reformable material includesholding a volume of particles inside a container having a firstelastomeric membrane surface; and infusing the volume of particles witha liquid; agitating the liquid to provide one or more surges of liquidto mobilize the volume of particles; and pressing a master shape intothe membrane with atmospheric pressure.

Implementations of the above method may include one or more of thefollowing. The method may provide locally distributed surges or globallydistributed surges. The surges can exert differential liquid forces onparticles to displace them relative to one another and facilitate theirmovement into a closely-packed volume. A differential pressure can beapplied between a master shape side and a liquid-particle side of themembrane. The pressure between a vacuum cap and the membrane can bedecreased to move the membrane in a first direction or increased to movethe membrane in a second direction. The membrane is free to moverelative to the master shape. Excess liquid can be removed to leaveparticles against the membrane. Air can be evacuated from space betweenthe membranes. The particles can be packed against the membranes and themaster shape. The liquid with the vacuum cap and membrane pressedagainst the master shape can pack the particles against the membranesand the master shape. The agitating operation can include pulsing orvibrating the liquid. The vibration frequency can be adjusted todisplace one particle relative to another to keep the particles movingfreely in relation to one another. The amplitude of the liquid pulsationcan be proximally equal to a diameter of the particles. A first surge ofliquid can be directed towards a desired transport direction and asecond surge smaller than the first surge can be directed in an oppositedirection to the transport direction. The agitating of the liquid can beused to minimize blockage. The method includes maintaining the volume ofthe container constant and completely filled to force the particlesagainst the master shape. The method includes extracting transitionalliquid from the container; and adding new liquid equal in volume to thetransition liquid.

In yet another aspect, a shape-reformable composition includes a carriermedium having a carrier density; and a plurality of solid bodies havinga density substantially similar to the carrier density, said solidbodies being transitionable from a formable state to a three dimensionalsolid shape. The bodies can have a density substantially lighter orheavier than that of the carrier if they have a high ratio of surfacearea to volume. The bodies can be stiff, flexible or elastomeric. Thebodies can be regular or irregular and can be of substantially differenttypes intermixed.

Implementations of the composition can include one or more of thefollowing. The carrier medium fills voids or interstices between thesolid bodies such that the voids or interstices are free of air or gasbubbles. The solid bodies can have near-liquid or fluent mobility duringthe formable state. The solid bodies can transition to the solid shapethrough an introduction and an extraction of a predetermined amount ofthe carrier medium. The solid bodies can be positioned in a containerhaving a first elastomeric membrane surface. Liquid can be introduced tomobilize the volume of particles. A master shape can be pressed into themembrane with atmospheric pressure. The resulting solid shape is astable, force-resisting shape. The solid bodies and carrier medium forma reversible state-changeable mixture. The carrier medium can be aliquid or a gaseous froth. The shape can be a reformable mold or areusable template to capture dimensions of impressed shapes for transferto a mold.

In other aspects, a system is disclosed for holding a volume ofparticulate material inside an air and liquid-impermeable container withat least one elastomeric membrane surface; deaerating the volume;infusing the volume with a liquid to cause it to be mobile; pressing amaster shape into the membrane via atmospheric pressure; and extractingthe liquid through one or more screen elements which are placed in oradjacent to the particle volume. The extraction causes atmosphericpressure to press the particles against the contours of the shape andagainst each other. This pressure continues to hold the particles inplace against the elastomeric membrane when the master shape is removedfrom the outer surface of the membrane. The system further has a meansto heat and drive liquid from the particle volume and, in certainembodiments, to leave a residue of binding adhesive which locks theparticles into a continuous force-resisting mass.

Operation of one embodiment is as follows with a particular embodimentof the container which has a rigid outside frame and a membrane face onthe top and bottom surfaces. With the particle volume infused by liquid,a master shape is pressed against the top elastomeric membrane of thecontainer by atmospheric pressure, thereby causing the shape to impressa complementary shape in the membrane. This pressing is accomplishedthrough use of a flexible or elastomeric vacuum cap which is sealed overthe shape and against the container's top surface membrane, followingwhich air is evacuated from between the top membrane and the vacuum cap.Liquid is then extracted from the volume and the particles within thecontainer are pressed together by atmospheric force which acts on allexterior surfaces of the tool-bed but in particular in opposeddirections against the vacuum cap and the bottom surface membrane. Airis then introduced into the vacuum cap, the cap removed and the mastershape removed from the formed surface of the elastomeric membrane.

In another embodiment, the container is formed against a master shapewith the process of liquid infusion, a pressing action via atmosphericpressure and a liquid extraction process. This embodiment is essentiallya flat envelope with a flexible outside rim and two opposed elastomericmembranes. To use this embodiment a master shape is placed on anair-impermeable surface, a membrane of the container is placed over theshape, and either a vacuum cap or a vacuum-bagging film is placed overthe container to effect forming of the elastomeric membrane against themaster shape. The envelope may also have a vacuum seal on its perimeterand so has the combined function of containing a mass of particles andof serving to extract air from between the master shape and theenvelope.

In implementations, there can also be a combined use of the first andsecond containers described above. A master shape may be placed on thetop elastomeric surface of the first rigid-framed container and then amembrane surface of the second container is placed over the shape. Thesecond container fits inside the frame of the first container and avacuum cap is placed over and sealed outside the second containeragainst the surface membrane of the first container. When the volumeunder the vacuum cap is evacuated the master shape is pressed betweenthe elastomeric sides or faces of the two containers. Liquid is thenextracted so that the two volumes of particles are pressed together andagainst the membranes surrounding the contained shape; the vacuum cap isvented with air and removed; the top container is removed; and the shapeis removed from the membrane of the bottom container. When the topcontainer is again placed over the bottom container, a closed, shapedcavity is formed which is complementary to the entire surface of themaster shape which was used to form the cavity.

In yet another embodiment, a combination of containers can be used inwhich two identical containers of either the first or the second typemay be pressed together around a master shape without use of the vacuumcap. In this case the containers are joined and sealed by either a sealmounted on one or both of the containers or by seals mounted on a sealring which fits between the two containers. The seal ring may be furtheremployed to channel vacuum or air pressure between the two containersand to hold the master shape in a precise orientation and positionbetween the two opposed containers. The seal ring may also furnishaccess to the formed cavity for the purpose of injecting a moldablematerial into the cavity.

In yet another embodiment of the container the container itself isformed into a replica of a master shape, or into a shape complimentaryto a master cavity by another combination of the elements and processesdescribed above. The exterior of this third type of container may beformed entirely from an elastomeric material or may be formed from acombination of elastomeric, flexible and rigid materials. Though thecontainer might be shaped against a single surface, it can also beshaped over substantially its entire surface by confining it within amaster cavity formed by two or more closely-fitting mold parts. Key tothis forming process is an expansion means within the third containerwhich presses the particulate material against the cavity walls.

In another embodiment, there is combined use of the containers whichemploy the three types of containers described above for a singlepurpose. The first or second types can be used to form a complementarycavity from a master shape. The third type of container can then beplaced in the cavity, which is now used as a master cavity, and thethird type formed complementary to the master cavity contours, therebycreating a replica of the original master shape.

It can be appreciated that there are numerous variations of containersand varied combinations of containers which can be employed either toform a surface which is complementary to the exterior surface of amaster shape in part or in whole, or to form a surface or surfacescomplementary to the interior contours of a hollow master shape ormaster cavity. For instance more than one container of the first type(rigid frame) or second type (flexible-edge) can be employed to form acontinuous surface complementary to a master shape's surface, with theelastomeric membranes of the containers either overlapping or beingabutted together. Containers of the second type may also have a membraneand particle configuration that allows two or more of the containers tobe “tiled” together to form a continuous surface of particle-backedmembranes. Likewise two or more containers of the third type can beemployed together to form a shape complementary to the interior of amaster cavity.

In yet other embodiments, a forming system also includes a base stationwhich provides evacuation of air, liquid infusion into and liquidextraction from the particle filled containers. The base station alsofurnishes vacuum forces to enable the forming operations to be performedon the various containers either singly or in combination. The basestation comprises a liquid receiver; onboard vacuum system or provisionto connect to an external vacuum source; an air compressor or provisionfor external connection to pressurized air; valves, fittings and tubingor piping to provide vacuum and to control the flow of liquids to andfrom the containers; an electrical power supply to operate the valves,process sensors and any onboard mechanical vacuum pumps and aircompressors; and a menu-driven process controller to operate the basestation.

In another embodiment, a forming system includes a heater which may beused to vaporize and drive out liquid from the particle filledcontainers, and further to heat any materials which may be used torecreate the contours of the original master shape through moldingagainst the complementary surface of the formed elastomeric membrane.The vaporizing or drying process is especially advantageous when theliquid contains a soluble binder which remains on the pressed-togetherparticles and locks the shaped volume of particles in place when theliquid has been driven out of the container. The heater may takenumerous forms to include a radiant heater, a convective air heater,heating elements within the particle-filled container, and various typesof inductive (e.g., microwave or radio-frequency) heaters. The heatermay be powered and controlled by the base station and its controller, orthe heater may be powered and controlled separately.

Next a reformable footwear making embodiment is detailed. In thissystem, the 3D model of the footwear as customized by the user or adoctor for the user is provided to a reformable shape object fabricator1006, which is detailed next. The fabricator renders a physical model ofthe 3D model and then applies a state-changeable mixture that includesuniform, generally ordered, closely-spaced solid bodies and a liquidcarrier medium, with the liquid filling any voids or interstices betweenthe bodies and excluding air or gas bubbles from the mixture. Within themixture, the solid bodies can be caused to transition from a near-liquidor fluent condition of mobility to a stable, force-resisting condition.To create mobility, a small excess quantity or transition liquid isintroduced to create a fluent condition by providing a slight clearancebetween the bodies which permits the gently-forced introduction of atleast two simultaneous slip planes between ordered bulk masses of thebodies at any point in the mixture. Transition to the stable conditionis caused by extraction of the transition liquid, removing the clearancebetween bodies and causing them to make stable, consolidated contact.

The 3D shape generator to generate the insole or sole is a completecomputer actuated system that is enclosed in the object fabricator. CADdata is downloaded by wire or wireless connection to the shapegenerator. Based on the desired dimensions, one embodiment of the 3Dshape generator forms a 3D object by having an array of computercontrolled moveable pins whose height is adjusted in accordance with theCAD design file, and the overall shape is smoothed by a Lycra sheet orfelt sheet. The pins or rods lift the felt or Lycra sheet to form a 3Dobject based on the CAD design file. In this embodiment, an array of N×Nmicro hydraulic actuators can be used to form the shape. This embodimentis a dense hydraulic planar pin-rod matrix array. Another embodimentactuates an N×N pin-rod matrix driven by servomotors. In either case,each pin-rod is controlled individually, similar to pixels on a screenexcept that the pixel has height as well.

In one embodiment, the N×N matrix can be an array of electro-mechanicalpins positioned in a frame. The frame is adapted to hold the pluralityof pins in a parallel position to one another in a series of columns androws, such that the distal ends of the plurality of pins together form aflat virtual plane. Each pin of the plurality of pins includes anelongated housing member defining a linear axis there through, and a pinmember adapted to slide linearly in either direction along the axis.Each of the housing members includes an upper electromagnet, and a lowerelectromagnet separated from the upper electromagnet. Each of theelectromagnet is adapted to move its respective pin member linearly ineither direction. Each of the pin member includes a linearpotentiometer, a, magnet and an electronic transmitter attached to anopposite end to the distal end, such that when each of the pin membersare moved linearly each respective linear potentiometer sends a signalto its respective transmitter which in turn sends an electronic signaldescribing its movement within its respective housing member, aplurality of electronic wires respectively connected to eachtransmitter, such that electronic signals can be relayed to and fromeach respective pin; an analog-digital converter connected to theplurality of electronic wires and adapted to convert the analogelectronic signals relayed by the transmitters into digital format to betransmitted, processed, stored, and then converted back into analog formfor return transmittal to the set of pins. A processor is connected tothe converter and adapted to retrieve the electronic signals from theconverter, store them, and retransmit them back to the converter whendesired, such that a user can displace the pin members from the virtualplane in any pattern, have electronic signals sent, processed, stored,and returned to the same set of pins, or another separate set of pins,at a later time to thereby displace the pins to the same positions asthe original pattern chosen by the user.

In one embodiment, the pin array device has each of the housing memberof each pin comprise an upper frame upper electromagnet, upper spring,lower electromagnet, lower spring and shield along the entire upperframe wall to separate magnetic field between each interactive pin. Thelower frame consists of the outer fixed part of the potentiometer andelectronic transmission from electronic transmitter to bothelectromagnets. The pin consists of a magnet, a mobile portion of thepotentiometer, electronic transmitter that picks up all the wire andsends position signal and feeds the power to both electromagnets via thelower housing. The electronic signal may be a Pulse Width Modulationsignal, and the displacement of each of the pin members is proportionalto the strength of the Pulse Width Modulation signal received by theelectromagnets.

While FIGS. 1F-1G show actuators or pins at predetermined locations forinsole use, the pins can be part of a grid array for adjusting arbitrarydimensions. In one embodiment, the pins are moved by the action of aplate, common to all or a portion of the pins, that can extend andretract along a single axis of motion. A clutch mechanism cooperateswith the moving plate to fix the pins at a desired position. In anexemplary embodiment, the shape generator 1004 can include a membranecovering the pins. A plurality of pins 1011-1018 arranged in an arraysuch that respective head portions 1021-1028 associated with the pinscollectively define a surface 1030. It will be appreciated that the areaof array is not necessarily defined by two Cartesian dimensions. Forexample, the pins could be arranged along a spherical or hemisphericalsurface, with the array spanning the azimuthal and polar dimensionsacross the surface of the sphere. The position of a given pin (e.g.,1011) can be adjusted along an axis of motion.

In one embodiment, an optional motion plate 1032 can be provided to movethe pins along the axis of motion as to adjust the position of the pins.The motion plate 1032 can be moved by reasonable mechanical orelectromagnetic means. For example, the plate 1032 can be moved via anelectrical motor, a hydraulic assembly, or one or more solenoid coilsexerting a magnetic force.

A clutch mechanism 1034 is operative to arrest the motion of a given pinat a desired position. The respective positions of the pins can beselected to deform the display surface into a desired raised image. Theclutch mechanism can comprise reasonable means for selectively arrestingthe motion of the pins. For example, the clutch mechanism 1034 cancomprise components for mechanically or magnetically engaging the pins.

One embodiment provides an upper plate with a plurality of aperturesthrough which corresponding pins forming the object's surface can pass.The pins can include head portions with areas larger than that of theirrespective apertures, to more fully tessellate the display surface andto help maintain the pins within the apertures. The upper plate canhouse part or all of a clutch mechanism that selectively engages one ormore pins to maintain the pins at a desired position. The upper platehouses one or more banks of solenoids that shift the position of one ormore portions of the clutch (not shown) that physically communicate withthe pins. In an exemplary embodiment, the solenoids shift the positionof one or more bars such that they contact or release circumferentialgrooves on the surface of the pins. This embodiment also provides alower plate and a base plate disposed parallel to the upper plate alongone or more support posts. A lifting plate can be suspended between thelower plate and the base plate on one or more guide posts. The liftingplate can be raised or lowered via a motor and belt system to adjust theposition of the pins. For example, the pins can be reset to a fullyraised position by raising the lifting plate to its maximum height. Themovement of the guide pins and the action of the clutch mechanism can beregulated by a processor.

As shown in FIG. 1I, two facing and opposite bed of pins 2210-2212 canform a 3D shape for the sole or insert. The insert and/or the shoe canbe produced in discrete sizes such as US sizes 4, 4.5, 5, 5.5, 6, 6.5,7, 7.5, 8, 8.5, 9, 9.5, 10, 10.5, 11, 11.5, 12, 12.5, 13, 13.5, 14,14.5, 15, 15.5, 16, 16.5, 17, 17.5, and 18, for example. Thus, aplurality of sized beds can be used, or one large pair of beds coveringsize 20 can be used to produce all other smaller sizes. Turning back toFIG. 1H showing one of the beds 2210-2212, the selected view of the 3Dobject creator comprises one row of four pins 2102-2108. It will beappreciated that a functioning computer controlled 3D object creator cancontain a large number of pins arranged across multiple rows in order toreproduce the shape of the 3D object with high fidelity.

In an exemplary embodiment, the rows containing the pins 2102-2108 arestaggered as to form a honeycomb pattern. Accordingly, the pins2102-2108 are arranged in a plurality of linear rows and one or morestaggered columns. Alternatively, the pins can be arranged in aCartesian grid, such that both the rows and the columns are linear. Itwill be appreciated that other methods of arranging the pins can beutilized, and that the placement of the pins will vary with thenecessary size and spacing of the pins, as well as the desired shape(e.g., flat, spherical, recessed) of the array.

In the illustrated display, the pins 2102-2108 have respective capportions 2112-2118 that define a raised surface. The cap portions2112-2118 can be covered by an elastic membrane or felt layer 2120 toprovide a relatively smooth surface for the object. The use of the pincaps 2112-2118 and the membrane 2120 will depend on the application. Thepins 2102-2108 pass through respective apertures in a stationary, outerplate 2124. The outer plate 2124 houses a clutch mechanism 2126 thatacts to maintain the pins in their desired positions. In an exemplaryimplementation, the clutch mechanism 2126 can comprise a series of rowbars and column bars having two associated positions. In a first, open,position, a given bar allows the pins within its associated row orcolumn to move freely. In a second, restraining, position, the bar ismoved to physically contact the pins at one of a plurality of evenlyspaced grooves on the pin, maintaining the pin at its position. Thespacing of the grooves corresponds to a desired resolution of thedisplay 2100. The position of the bars can be changed via one or morebanks of solenoids. In an exemplary embodiment, the bars are biased, bya spring or similar mechanism, to remain in the restraining position,until a solenoid is actuated to move the bar into an open position.

During operation, the pins can be reset into a fully extended positionby a reset plate 2130. The reset plate 2130 can then be incrementallywithdrawn to allow the pins 2102-2108 to retract toward the interior ofthe display device. In an exemplary embodiment, the reset plate 2130 ismoved by a motor and belt arrangement. The pins 2102-108 have associatedsprings 2132-2138, with each spring (e.g., 2132) attached at a first endto the underside of the outer plate 2124 and at a second end to the endof the pin (e.g., 2102) opposite the cap portion (e.g., 2112). When thepins 2102-2108 are fully extended, the springs 2132-2138 are compressedagainst the underside of the outer plate 2124. The springs 2132-2138thus provide a tensive force on the pins 2102-2108 as to draw the pinstoward the interior of the object being formed.

The movement of the reset plate 2130 and the operation of the clutchmechanism can be coordinated by a controller 2140 to adjust the positionof the pins 2102-2108. The controller 2140 can provide informationrelating to the desired pin positions to the projector. The reset plate130 can be incrementally withdrawn toward the interior of the object. Inan exemplary embodiment, the reset plate 2130 withdraws in incrementsequal to the spacing between the grooves on the pins 2102-2108. Aftereach retraction of the plate, the clutch mechanism 2126 can beselectively activated to release one or more of the pins, while leavingothers secured. The tensive force provided by the springs 2132-2138pulls the ends of the released pins flush against the reset plate 130,such that the released pins retract to a uniform level defined by theposition of the reset plate. The secured pins remain at their previouslevel. The pins are then resecured by the clutch mechanism, and theplate is retracted by another increment. This process is repeated as thereset plate 2130 retracts to leave each pin at a desired level ofextension.

In another embodiment, the pins pass through respective apertures in astationary, outer plate housing a first portion of a clutch mechanismthat acts to adjust the pins into desired positions. In an exemplaryimplementation, the first clutch portion can be piezoelectric restraintsfor the plurality of pins. In a default position, a given restraintloops around its associated pin, but allows the pin to move freely. Uponthe application of an electrical current, the restraint contracts as tophysically contact its associated pin at one of a plurality of evenlyspaced grooves on the pin. This fixes the pin to the outer plate,maintaining the pin at a stationary position. The spacing of the groovescorresponds to a desired resolution of the 3D object being formed. Thepins also pass through respective apertures in a moving plate which canbe moved by a motor and belt arrangement. The moving plate houses asecond portion of the clutch mechanism with piezoelectric restraints forthe plurality of pins. The movement of the moving plate and theoperation of the first/second clutch portions can be coordinated by acontroller to adjust the position of the pins. The moving plateoscillates in a direction normal to the outer plate and a base platebetween a first position, closest to the base plate and a secondposition, closest to the outer plate. In an exemplary embodiment, thefirst position and the second position are separated by a distance equalto the spacing between adjacent grooves. The pins begin in a defaultposition, fixed to the outer plate by the first clutch portion. In anexemplary embodiment, the default position of the pins is a fullywithdrawn position (e.g., the first clutch portion engages the uppermostgroove of each pin). Since the default position of the pins is known,the controller can determine the distance between the default positionand a desired position as a number of increments, as defined by thegroove spacing of the pins. The controller can thus select one or morepins to extend by one or more increments. While the moving plate is inits first position, the selected pins are released by the first clutchportion. Simultaneously, the second clutch portion engages the selectedpins, such that the pins are fixed to the moving plate. The moving platecan then be moved to its second position. Once the plate reaches thesecond position, the second clutch portion releases the selected pins,while the first clutch portion reengages the pins. It will beappreciated that the motion of the moving plate can be controlled by thecontroller such that the first clutch portion can engage the pins at agroove one increment below the default position. Accordingly, theselected pins are extended by one increment. This can be repeated anumber of times, to allow one or more pins to be moved to a desiredposition up to a maximum extension. The final position of each pin willbe determined by the number of times the first and second clutchportions are activated for the pin. This can be controlled by thecontroller according to the desired position of the pin. Once the pinshave been positioned, the controller can direct the object fabricator1006 to copy the 3D object formed by the pin grid 3D shape generator.

In another exemplary clutch mechanism, a pin can be encased in a solidrestraining material having a low melting point. For example, therestraining material can be an alloy of lead and one or more othermetals. The restraining material is contained in a container having arelatively high melting point. The clutch mechanism disengages byapplying heat from a heat source to the restraining material in order tobring it to a liquid state. The heat source can be applied by a laserapparatus (not shown) directed on the restraining material or by aheating element associated with the container. In an exemplaryimplementation, the container is the heat source, producing resistiveheat upon the application of an electrical current. While therestraining material is in a liquid state, the pin can move freelythrough the aperture. Once the heat source is deactivated, therestraining material cools and returns to a solid state, restraining thepin.

In yet another exemplary clutch mechanism, a wire has shape memoryproperties is looped around a pin. The material with shape memoryproperties has the ability to return to an imprinted shape when heated.A desired shape can be imprinted into the material by molding thematerial at a high temperature and maintaining the desired shape as itcools. Below a threshold temperature, the material is relativelyflexible and can be deformed away from the imprinted shape with relativeease. Once the material is heated above the threshold temperature,however, it reverts back to the imprinted shape with some force. In anexemplary implementation, the wire is a formed from nitinol, an alloy ofnickel and titanium. The wire is shaped such that the loop is openedaround the pin and the pin can move freely through the loop. A currentcan be applied to the wire to heat the wire via resistive heating to atemperature greater than its threshold temperature. This causes the wireto return to its imprinted shape, engaging the pin as the loop closes.The wire returns to its imprinted shape somewhat forcefully, such thatthe tensive force on the ends of the wire is insufficient to restrainit. In an exemplary embodiment, the wire is looped around a groove inthe surface of the pin to facilitate engagement of the pin. When thecurrent is no longer applied, the wire 352 cools and returns to its moremalleable state. Once the wire cools below threshold, the tensive forceapplied can once again deform the wire into an open shape, releasing thepin.

FIGS. 1J, 1K, 1L, and 1M show a first container embodiment, a mastershape and a vacuum cap, and further show a sequence of operations tocreate a shaped impression, complementary to the master shape, in thesurface of one elastomeric membrane face of the container. Turning nowto FIG. 1J, a container 5 is shown with a rigid container frame 10 andelastomeric top and bottom membranes 20 and 25, resting on a base 13which separates the bottom membrane 25 from contact with any surfacethat the base 13 and the container 5 rest on. The top membrane 20 isbonded to a perimeter frame 17 so as to have an air-tight interfacebetween the container frame 10 and the membrane 20. The container frame10 is affixed to a continuous vacuum-activated seal 30 which is bondedto the container frame 10. The seal 30 is resilient and acts much like asuction cup to hold the perimeter frame 17 to the container frame 10.The bottom membrane 25 is bonded directly to rigid container frame 10since the membrane 25 is not a working surface wearer to damage, incontrast to the working surface of membrane 20 which is subject todamage. In one embodiment, the bottom membrane 25 can be affixed by aperimeter frame and vacuum seal as described above. In yet anotherembodiment with more complexity, mechanical clamps and a pressure sealcan be employed to affix either top or bottom membranes. Tubes 40, 50and 60 penetrate a toolbed or a container frame 10. The tube 40communicates with a seal 30 through an opening 45, and the seal 30affixes the membrane 20 to the container 5 by a vacuum (indicated byarrow 43) acting through the tube 40. The vacuum seal 30 can beinactivated by introducing air through the tube 40, allowing themembrane 20 and the frame 10 to be removed in order to insert or removea volume of particles from the container 5, or to replace a damagedmembrane 20 or internal screen element. The tube 50 communicates with amain particle screen 55 which is overlaid with a volume of particles 80.Arrow 53 indicates the flow of liquid into the particle volume throughscreens 55. The particle screens 55 serve to hold all particles in thecontainer 5 while allowing liquid to flow in and out of the particlemass. There is a double layer construction of both screens 55 with thetubes 50 and 60 communicating between the layers. The particles cannotpenetrate the outer layers of the screens and so do not move into thetubes as air is evacuated or liquid extracted. Detail 57 showsextensions of tubing 50 penetrating into the center of thedouble-layered screens. The extensions have perforations that enabledistributed liquid flow along the length of the tube inside the screen.The tube 60 communicates with a rim evacuation screen element 65 whichfollows the entire inside upper perimeter of frame 10 and is likewiseperforated along its length within element 65. Arrow 63 points outwardto indicate deaerating vacuum force acting on the container volume viathe evacuation element.

Turning now to the top of FIG. 1J, a vacuum cap 90 is shown with acontinuous flexible or elastomeric membrane 95 bonded to anotherperimeter frame 100, the frame also having a continuous vacuum-activatedseal 105 bonded to the frame 100. The seal 105 is identical in designand function to the seal 30. The vacuum cap 90 has a tube 110, whichcommunicates with the vacuum seal through an opening 115, and a tube 125which in turn communicates with the underside of membrane 95 through aport 120.

A master shape 130 is shown resting on membrane 20. The master shapewill used to form a shaped impression in the membrane as described next.To prepare for the forming process, a membrane 20 is sealed to thecontainer; air is removed from the volume of particles as shown by arrow63; and liquid is introduced into the particle volume as shown by arrow53. Liquid flow is cut off when there is sufficient liquid to allowparticles to move in relation to adjacent particles as displacing forceis exerted on either the top or bottom membrane of the container.

FIG. 1K shows a side view of the container frame 10 with a vacuum cap 70resting over the master shape 130 prior to being sealed against themembrane perimeter frame 15 to which the membrane 20 is bonded, with themembrane affixed using the seal 30 to the container frame 10. The master130 is resting on the unformed surface of membrane 20 with the movableparticles between membranes 20 and 25.

FIG. 1L shows a cutaway view with the vacuum cap 90 affixed by the seal105 against the perimeter frame 15 by vacuum through the tube 110 asshown by an arrow 113. In addition the space between the vacuum capmembrane 95 and the top membrane 20 has been evacuated through the tube125 as shown by an arrow 127. The vacuum cap membrane 95 is pressed downagainst the master shape 130 and against the surface membrane 20 byatmospheric pressure which also acts opposedly against container bottommembrane 25. Liquid is then extracted by a pump or vacuum from theparticle volume through a tube (not shown) through the particle screen55, causing atmospheric forces acting on bottom membrane 25 to pack theparticles against top membrane 20 which is forced against the mastershape since air has been evacuated from between the vacuum cap membraneand top membrane 20. Any leakage of air into the container, which wouldadd atmospheric pressure back to the container and so reduce the packingforce on the particles, can be removed by continuing vacuum extractionof liquid through particle screens 55 or by vacuum extraction throughthe perimeter evacuation screen element 65.

When the master shape 130 is removed from the surface of the membrane20, an impressed shape 135 remains which is complementary to the shape130. The differential pressure on the container by vacuum extraction iscontinued, thereby maintaining opposed atmospheric forces that act tokeep membranes 20 and 25 pressed against the particles and soimmobilizing them to keep the impressed shape stabilized. In form theseal is a continuous channel with the legs angled outward. The channelhas a single opening and a vacuum and vent tube connected to it asdescribed with reference to FIG. 3A. The material of the seal isresilient since the legs will be pressed against a surface and mustconform to and seal against the surface. The legs are separated by asufficient distance that they will be pressed into contact with thesurface by atmospheric pressure with a greater force per unit area thanatmospheric pressure. In function, when the legs of the channel arepressed against a smooth surface and the vacuum introduced inside thechannel, the seal legs deform against the surface and the deformed areais substantially less than the area inside the channel. In experiments aratio of deformed area to inside area of 1 to 2 has been shown to bevery effective in sealing against a smooth surface if the durometer ofthe seal's elastomeric material is around 40. In operation the seal issimply placed against or gently pressed against a smooth air-impermeablesurface. A vacuum is introduced through the tube, extracting air fromwithin the seal and so enabling atmospheric pressure to force the sealagainst the surface. Any leakage from atmosphere outside the seal isscavenged by the vacuum and so does not enter the volume inside theperimeter of the seal even if a full vacuum is imposed on that volume.To release the seal air is introduced via the tube or a small blade canbe slipped between the seal and surface to break the internal vacuum.

The particles can be a reversible state-changeable mixture having aplurality of solid bodies and a carrier medium, with the carrier mediumfilling any voids or interstices between the bodies. Within the mixture,the solid bodies can be caused to transition from a formable state,preferably a near-liquid or fluent condition of mobility, to a stable,force-resisting condition through introduction and then extraction of aslight excess quantity of the carrier medium beyond that required tofill the interstices of the bodies when closely packed. In mostembodiments, the carrier medium is a liquid preferably excluding any airor other gases from the mixture, and most of the discussion will revolvearound such embodiments. However, some embodiments use a carrier mediumthat is a liquid-gas froth.

The mixture can be rapidly shifted from a formable (preferablynear-liquid or fluent) state to a stable force-resisting state and backagain to the formable state, through slightly altering the carrier-solidproportions of the mixture, and the system further provides methods andapparatus for using the mixture. Embodiments are characterized by one ormore of the following advantages: the ability to pressurize a mixtureand drive it against a complex surface as if it were a liquid; theability to create a “near-net” or extremely accurate representation of ashape due to the negligible volumetric change that accompanies a statechange; the ability to effect the state-change with a very small volumeof single-constituent transfer and with consequently small actuationdevices without the need for a vacuum pump, without chemical reactions,and with no need for thermal or electrical energy to be applied to themixture; the ability to greatly alter the volume of any elastic orotherwise dimensionally changeable container, envelope or chamberthrough the free-flowing transfer of the mixture from one container toanother; and the ability to tailor the mixture to satisfy a wide varietyof physical specifications in either the flowable or the stable state.

The mixture can be used in reformable molds or other shaping tools, andin reusable templates that capture the dimensions of impressed shapesfor transfer to a mold. The mixture can also be used in any product orshape that benefits from the incorporation of arbitrary reformability orprecise reconfigurability. The mixtures further provide usefulproperties for use in a wide range of shock-absorbing, leveling,protective and supportive elements or apparatus.

The mixture in its formable state may be loosely compared to quicksand,while the mixture in its stable state may resemble hard-packed sand oreven cement, with the transition being caused by the transfer of arelatively small amount of liquid. Hence the mixture, while in theformable state, includes enough liquid to fill the interstices betweenthe nested solid bodies, and an excess amount of liquid that is referredto as the transition liquid. In the stable state the transition liquidis absent and the bodies are completely packed or nested.

In preferred embodiments the solid bodies are uniform, generallyordered, and closely spaced, with the predominate mass of the bodiesclose-packed and touching. To create mobility, the transition liquid isintroduced in just-sufficient quantity to create a fluent condition byproviding a clearance between some of the bodies, which clearancepermits the introduction of at least two simultaneous slip planesbetween ordered masses of the bodies at any point in the mixture. Thebodies themselves separate freely from one another under movement of theliquid and without turbulent mixing, and shift relative to one anothergenerally in ordered bulk masses. The bodies should be of a density thatis close enough to that of the liquid to permit flow of the bodies alongwith the liquid, or should have a size or structure that facilitatesmovement of the bodies along with the liquid.

In an embodiment, the surface of the mixture while in the formable stateis first made to conform to a desired shape. The bodies in the mixtureare then caused to transition from the fluent condition to the stablecondition through extraction of the transition liquid. This extractionremoves the clearances required to provide slip-planes between orderedmasses of the solid bodies, thereby causing the bodies to make nested,packed, interlocking or otherwise stable consolidated contact. Themixture, now in the stable state, has a surface that conforms to thedesired shape.

The mixture can be used in molds, templates or other products throughholding the mixture in, or transferring quantities of the mixture whilein the fluent condition into and out of variable-contour orvariable-volume containers or chambers. The mixture can be stabilized byremoval of the transition liquid, which may cause an elastic membrane tobe pushed against the consolidated bodies by ambient pressure, or bytransition liquid removal that causes the solid bodies to pack togetherunder liquid tensile forces, thereby creating an ordered,deformation-resisting structure through surface friction or throughsurface adhesion of one body to another.

In certain embodiments, the mixture can be held inside a container ortransported into a container with a flexible, elastically deformable andstretchable wall. The process then extracts the transition liquid fromthe mixture so as to cause body-to-body contact and force-resistingstability through pressure external to the container acting on theconfined, ordered, abutting bodies. Transfer of fluent mixture into andout of the containers, or displacement of mixture within the containerscan be accomplished by pressure forces within the mixture, with theseforces being distributed uniformly throughout the mixture by the liquidcarrier medium.

This distribution of uniform pressure against the surface of each body,coupled with the clearance volume furnished by the transition liquid,assures that the bodies are not forced against one another while themixture is in the fluent condition. This elimination of body-to-bodycompression forces in turn prevents the bodies from sticking togetherand resisting displacement while the mixture is in the fluent condition.Pressure forces in the liquid can be exerted through pressing a shapeagainst an elastic, stretchable membrane that constitutes at least onesurface of a chamber substantially filled with the fluent mixture, orsuch forces within the liquid medium of the fluent mixture may beinduced by a two-way pump or other transfer system.

The bodies themselves may have various geometries and may be providedwithin a state-change mixture in one uniform type, or there may be twoor more types or sizes of bodies dispersed or layered within a mixture.For example spherical bodies of one size might have smaller bodiesfilling the interstices between the larger bodies, or a layer of shortfiber bodies might float above a layer of spherical bodies. Flake-likebodies can be also be used, in which case the flat faces of the bodiescan be pressed against one another to create a force-resisting bodymass. The flat faces provide many times the contact area of abuttingspheres, with accordingly higher friction or adhesion potential whenconsolidated against one another. If the flakes are in the form of alaminate that has one side heavier than the carrier medium and one sidelighter, and if the flakes are closely spaced and in a medium whichsuppresses turbulence and solid body tumbling, the bodies will tend tobe supported in, and to be consolidated in, an ordered parallelconfiguration. In this case, as with the spherical bodies, thetransition liquid quantity will be just sufficient to create shearmotion of body masses under low displacement forces.

Mixtures with more than one type or size of body can be used with thebodies either intermingled or layered separately, as by differingdensities or the inability of bodies of one layer to pass through bodiesin the adjacent layer. Bodies of different sizes or types may also beseparated from one another by flexible or extensible porous materials orfabrications that allow passage of liquids but not of the confinedbodies.

The degree of accuracy or irregularity on the surface of a stabilizedmass of the mixture is dependent upon the relationship between thefineness of the bodies and the dimensions to be captured, a coveringmembrane's thickness and conformability, and the size and degree ofregular packing order of a state-change mixture's solid bodies. If thebodies are very small compared to the contours of a shape that is to bereplicated, or if the interstices between larger bodies in the mixtureare filled by such smaller bodies, the mobile solid bodies of themixture will consolidate and assume a near-net shape relative to anyimpressed shape when the transition liquid is extracted from themixture.

In additional embodiments, the mixtures are stored external to one ormore molds, tools or fixtures, and are selectively introduced,stabilized and made fluent again in the tools. Formulas of the mixturesor solid bodies and liquids of the mixtures may be stored separately,and may be mixed or separated as required for effective operation ofseparate elements of a forming or tooling system.

In yet other embodiments, flexible elements containing state-changemixtures are used to capture exterior or interior contours of a shapeand to transfer the contours to other state-change elements. Throughsuch “templating” operations a negative of a shape or surface may beproduced and then a shape or surface identical to the first may beproduced by forming the surface of a mixture against the transfertemplate. Individual elements might also be used to transfer portions ofone shape to another shape and so create variations that combine thecontours of two or more shapes into a single shape.

In still other embodiments, several elastic, extensible elements filledwith state-change mixtures slide freely upon one another and relative tothe contained mixtures in order to conform to highly contoured shapes.These embodiments would be used when the elastic stretch of a singlemembrane element is not sufficient to capture details of a shape.

Further embodiments include methods of displacing fluent mixtures withinvariable-volume flat elastic envelopes by pressing the envelopes againstshapes with exterior air or liquid pressures, or pressing with physicalelements such as bundles of rods or fingers that slide relative to oneanother. The pressing force pressurizes the liquid carrier medium andcauses the envelopes to extend and conform to the shapes as thecontained fluent mixtures flow within the envelopes under the uniformlydistributed pressure forces within the liquid. Embodiments alsocontemplate the creation of hollow voids within a mixture-containingenvelope, with the impressed shape causing the collapse of the voids sothat the mixture need not be pumped into and out of a chamber to permitcapture of a shape.

Yet other embodiments include methods for creating a sculptablecondition in specific state-change mixtures through placing the mixturesin a quasi-stable state. The solid bodies are held in contact byextraction of a portion of the transition liquid, yet have sufficientlubricity or low contact friction to be displaced relative to oneanother by externally imposed forces. The bodies can be displaced intovoids created within a mass of the quasi-consolidated mixture, or can beprogressively displaced along the surface of the mixture from one regionof the mass to another. In some embodiments, properties of flow of themixture and the resistance to deformation of the abutted bodies arepredetermined so as to be a function of the imposed external forces, andso to be subject to variable control that allows intermediatequasi-stable, sculptable or displaceable conditions within or on thesurface of the bulk mixture.

State-change mixtures may also use solid bodies along with astate-changeable liquid carrier medium. The method for changing themixture from fluent to stable and back again is, as described above,through transfer of a small amount of excess liquid; however, themixture can be further solidified by changing the state of the carriermedium from liquid to solid.

In yet another embodiment, a state-change mixture is consolidated withina mold chamber and the liquid carrier or a second liquid component iscirculated while held to a pressure below ambient. Through heating andcooling of the circulating liquid, the mold itself can be heated orcooled.

Still another embodiment of the state-change mixture has solid bodiesthat are hollow and very light, and a carrier medium comprising aliquid-gas froth of similar density. The froth is destroyed whenextracted since the gas within it expands and separates from the liquidcomponent; then the froth is reconstituted from the liquid and gas andreintroduced into the body mass to recreate a fluent mixture. The liquidcomponent of the froth may be a solvatable (solvent-releasable) adhesivethat can be dried to hold the consolidated bodies together and thenre-dissolved by the frothed carrier medium. Very light bodies can alsosurrounded by a denser liquid, with the mixture likewise becoming fluentand then stabilized with transfer of a small quantity of transitionliquid; however, the tendency of the bodies to adhere together undercontact pressure is preferably countered, or liquid-like transfer of themixture, especially through small lines or passages, becomes difficultif not impossible.

In additional flat envelope embodiments internal and external elementsimprove their functioning as lightweight tooling and templates. Includedare methods to support these mixture-containing envelope structures,both internally with flexible reinforcements and externally with tubular‘foot’ structures that also contain state-change mixtures. The flatenvelopes may also be backed or supported by liquids or dry media withthe ability to capture precise impressions of a shape with the abilityto be switched from a liquid-like state to a firm state, or even to afully hardened state that resembles concrete yet can be returned to aformable condition.

The state change from liquid-like to solid-like properties within themixtures is effected by the transfer of a small amount of excess carriermedium, the transition liquid, into and out of the mixtures. When thetransition liquid is present, preferably in just-sufficient quantity tocreate the degree of support and clearance that provides for at leasttwo slip-planes, the solid bodies have a degree of mobility similar tothat of the liquid medium of the mixture. The slip-plane condition ofmobility can be generated through very small liquid pressuredifferentials or through externally imposed forces that displace thecarrier liquid and the supported bodies along with the liquid. Orderedbulk masses of the bodies can shift relative to other ordered masses atany point within a continuous volume of the mixture, and the location ofthe slip-planes can fluidly shift under any slight differential forcetransferred from one body to another. It is preferred to preventfrictional contact between bodies during such force transfer by havingthe liquid medium of the mixture furnish a viscous or ‘streaming’resistance to contact, and also for the medium to furnish a degree ofbody-surface lubrication so that light body contacts do not createfriction between bodies.

Lubricity under high contact forces, as is required for many lubricatingmedia, is not necessary within the mixtures since the bodies are ineffect free-floating during flow, with any imposed liquid pressureforces being uniformly distributed against the surface of each body. Forexample a nearly ideal aqueous liquid medium can be formed by dissolvinga small quantity of a soluble long-chain polymer such as polyethyleneoxide into water. The medium carries solid bodies of a similar densitywithout turbulence and friction-producing contact, allows the bodies tomake non-lubricated surface contact when the medium is extracted, andcauses the bodies to readily separate when the transition liquid isreintroduced.

When the transition liquid is extracted so that the solid bodies are ina stable configuration with ordered, packed and consolidated contact,the degree of resistance to externally imposed forces depends on suchtailorable, engineered physical properties as body shape, bodyelasticity and compressibility, body surface properties of roughness,smoothness or natural molecular adhesion, residual adhesiveness orlubricity of the liquid medium on the contacting surfaces, surfacetension of the medium, and variations of liquid medium or bodyproperties with changes of temperature or pressure; alteration of theresistance properties through replacement of the first liquid with asecond liquid medium, rinsing of the bodies and the first medium with asecond or sequential liquid media, vapors or gaseous fluids; and anyother engineered variations in the bodies and first liquid medium, andin other sequential introductions of various fluids into the mixtures orthrough the consolidated bodies. Any adhesive or clinging contactbetween the bodies is preferably relieved through polar molecular actionof the first liquid medium, or through an intermediary treatment withother liquids or fluids prior to reintroduction of the first liquidmedium.

The container works with quickly reversible state-change mixtures whichcan be rapidly shifted from a near-liquid or fluent state to a stableforce-resisting state through slightly altering the liquid-solidproportions, and the system further provides methods and apparatus forutilizing the mixtures. Embodiments are characterized by one or more ofthe following advantages: the ability to pressurize a mixture and driveit against a complex surface as if it were a liquid; the ability tocreate a “near-net” or extremely accurate representation of a shape dueto the negligible volumetric change which accompanies a state change;the ability to effect the state-change with a very small volume ofsingle-constituent transfer and with consequently small actuationdevices, with a low-energy mechanical actuation, and without requiring avacuum pump, thermal, chemical or electrical energy to be applied to themixture; the ability to greatly alter the volume of any elastic orotherwise dimensionally changeable container, envelope or chamberthrough the free-flowing transfer of the nearly solid mixtures from onecontainer to another; and the ability to tailor the mixtures to satisfya wide variety of physical specifications in either the flowable or thestable state.

The mixtures can be employed in reformable molds or other shaping tools,and in reusable templates which capture the dimensions of impressedshapes for transfer to a mold. The mixtures can also be used in anyproduct or shape which benefits from the incorporation of arbitrarilyreformability or precise reconfigurability. The mixtures further provideuseful properties for but are not limited to application in a wide rangeof shock-absorbing, leveling, protective and supportive apparatus.

It can be appreciated that there are numerous variations of containersand varied combinations of containers which can be employed either toform a surface which is complementary to the exterior surface of amaster shape in part or in whole, or to form a surface or surfacescomplementary to the interior contours of a hollow master shape ormaster cavity. For instance more than one container of the first type(rigid frame) or second type (flexible-edge) can be employed to form acontinuous surface complementary to a master shape's surface, with theelastomeric membranes of the containers either overlapping or beingabutted together. Containers of the second type may also have a membraneand particle configuration that allows two or more of the containers tobe “tiled” together to form a continuous surface of particle-backedmembranes. Likewise two or more containers of the third type can beemployed together to form a shape complementary to the interior of amaster cavity. More details on the reformable manufacturing aredisclosed in commonly owned patents to Jacobson et al including U.S.Pat. No. 6,398,992 and Pub. No. 20050035477 and 20070187855, thecontents of which are incorporated by reference.

The footwear can be custom produced at the request of a customer, whocan specify the nature of the customization for one or more pairs offootwear. Each shoe of a pair of the footwear may have a differentdesign, message or message portion designed into it and rendered usingthe bed of pins described below to make the custom shoe design messagesor shapes, and then the bottom sole can be fabricated using thereformable bed described below. Once the negative is fixed in thereformable bed, suitable materials for the bottom sole can be depositedand cured and can include rubber, plastic, or foam. Furthercustomization can be done by a Computerized Numerical Control (CNC)where component design can be integrated with computer-aided design(CAD) and computer-aided manufacturing (CAM) programs. The device can beprogrammed to use a number of different tools-drills, saws, and so on.Alternatively a number of different machines can be used with anexternal controller and human or robotic operators that move thecomponent from machine to machine. Regardless, a series of steps neededto produce a part can produce a part that closely matches the originalCAD design in a highly automated fashion. In accordance with aspects ofthe subject matter disclosed herein through the use of reformable bedand a suitably programmed CNC tools, customized footwear with custom cutsole designs, can cost effectively be created in small quantities andyet scalable for mass-customization.

Composite Sole/Insole

The insole/sole can be a composite material. A composite material (alsocalled a composition material or shortened to composite) is a materialmade from two or more constituent materials with significantly differentphysical or chemical properties that, when combined, produce a materialwith characteristics different from the individual components. Theindividual components remain separate and distinct within the finishedstructure. The new material may be preferred for many reasons: commonexamples include materials which are stronger, lighter, or lessexpensive when compared to traditional materials.

FIG. 1N shows various medial side views of a spring element that isformed by using the composite material for strength. The spring elementcan be a leaf spring formed of a composite material which is aerospacegrade to provide resilience and long lasting spring. In one embodiment,the above sand printing system provides lightweight, lowthermal-capacity reformable tooling for high-performance compositeprocessing evaluation, parts prototyping, trial production and full-rateproduction of polymer matrix composites (PMCs) footwear components. Forexample, reinforcement can be glass fiber, carbon fiber, Kevlar fiber,natural fiber, ceramic fiber, particulate of nano materials. The resincan be polymer, metal, or ceramic.

Optimized tooling would provide a ramp-up rate consistent with theheatup rate of the composite material, while providing the strength towithstand the process temperatures and pressures. Going down the firstrow, spring element 51 consisting has a superior spring element 47including toe spring in the forefoot area 58 and an inferior springelement 50 including a compound curved shape forming a concavity 76 inthe midfoot area 67. Next is a spring element 51 having a superiorspring element 47 that is relatively flat in the forefoot area 58 and aninferior spring element 50 including a compound curved shape forming aconcavity 76 in the midfoot area 67. In the next row, spring element 51has a flexural axis 59 in the forefoot area 58 consisting of a superiorspring element 47 including toe spring and an inferior spring element 50including a relatively flat shape. Next in the row is a spring element51 having a flexural axis 59 in the forefoot area 58 consisting of asuperior spring element 47 having a relatively flat shape and also aninferior spring element 50 including a relatively flat shape. In thethird row, spring element 51 has a flexural axis 59 in the forefoot area67 consisting of a superior spring element 47 made in continuity with aninferior spring element 50 forming an elliptical shape on the posteriorside 34 and next to it is a spring element 51 having a flexural axis 59in the midfoot area 67 consisting of a superior spring element 47 formedin continuity with an inferior spring element 50 forming an upwardlyrounded shape on the posterior side 34. In the next row is a springelement 51 having a flexural axis 59 in the midfoot area 67 consistingof a superior spring element 47 formed in continuity with an inferiorspring element 50 forming a downwardly rounded shape on the posteriorside 34 and next to it is a spring element 51 having a flexural axis 59and a concavity 76 in the midfoot area 67 consisting of a superiorspring element 47 formed in continuity with an inferior spring element50 forming an elliptical shape on the posterior side 34. In the next rowis a spring element 51 consisting of a superior spring element 47, aposterior spacer 42, and an inferior spring element 50 having arelatively flat shape. As shown, a posterior spacer 42 can provide asubstantial elevation in the rearfoot area 68 and next to this is springelement 51 consisting of a superior spring element 47, a posteriorspacer 42, and an inferior spring element 50 having an upwardly curvedshape at the posterior side 34. As shown, a posterior spacer 42 canprovide a substantial elevation in the rearfoot area 68. In the next rowis a side view of a spring element 51 consisting of a superior springelement 47, a posterior spacer 42, and an inferior spring element 50having a complex curved shape at the posterior side 34. As shown, aposterior spacer 42 can provide a substantial elevation in the rearfootarea 68. To the right of that is a spring element 51 consisting of asuperior spring element 47, a posterior spacer 42, and an inferiorspring element 50 having an arcuate shape. As shown, a posterior spacer42 can provide a substantial elevation in the rearfoot area 68. In thenext row is a spring element 51 consisting of a superior spring element47, a posterior spacer 42, and an inferior spring element 50 that isorientated downward along the posterior spacer 42, but which isrelatively flat near the posterior side 34. As shown, a posterior spacer42 can provide a substantial elevation in the rearfoot area 68. To theright is spring element 51 consisting of a superior spring element 47made in continuity with an inferior spring element 50 forming anelliptical shape on the posterior side 34. As shown, the anteriorportion of the inferior spring element 50 is affixed to a posteriorspacer 42 which can provide substantial elevation in the rearfoot area68. Alternately, an inferior spring element 50 can be made as a separatepart, and can then be affixed to a posterior spacer 42 and/or superiorspring element 47 near the anterior end of the inferior spring element50, and also be affixed to the superior spring element 47 near theposterior end of the inferior spring element 50.

Alternatively, the system can form a multi-polyurethane insole for shoesby curing different elasticity polyurethane materials sequentially aboveeach other in the reformable mold formed above including: a lowelasticity polyurethane foam disposed on the top side of the insole; ahigh elasticity polyurethane foam and a mid elasticity polyurethane foamsequentially laminated at the inside of a generally oval concavedportion formed on the bottom surface of the insole abutting against awearer's heel portion; and a foot arch base formed integrally with theinsole 1 in such a manner as to be protruded from the middle portion ofthe insole. When the multi-elastic insole is disposed on the bottomsurface of the shoe, the wearer's weight is much collected on the ovalconcaved portion formed on the bottom surface of the insole abuttingagainst the wearer's heel portion on the low elasticity polyurethanefoam formed on the top side of the insole, and then, the wearer's bodypressure (foot pressure) is distributed by means of the low elasticitypolyurethane foam. Next, the minute movements of the wearer's musclesare caused by means of the high elasticity polyurethane foam formed atthe inside of the oval concaved portion 3, and the impacts are finallyabsorbed by means of the mid elasticity polyurethane foam 4 disposedbeneath the high elasticity polyurethane foam at the inside of the ovalconcaved portion 3, so that the increasing rate of the foot pressure canbe reduced. A foot arch base is formed on the top portion of themulti-elastic insole 1 serves to support the load of the wearer's footgenerated by the foot pressure, thereby greatly reducing the fatigue ofthe wearer's foot. When the shoes having the structure of a composite ora multi-elastic insole are worn, the high, mid and low elasticitypolyurethane foams are sequentially laminated on the bottom surface ofthe insole abutting against a wearer's heel portion, so that the impactsgenerated from the wearer's foot sole while working for long hours at astate of standing up on a hard floor are all absorbed, thereby makingthe wearer feel comfortable, which reduces the wearer's leg and footfatigue and prevents the increasing rate of the foot pressure.

In addition to the multi-elastic material and/or composite materialdiscussed above, the sole and the cushion can be made of rubber or canbe made of a thermoplastic resin. Preferable materials are those whichare easily thermoformable into desired flexible configurations.Materials which can be thermoset after molding and retain the flexiblecharacteristics for the sole components of the present system areincluded within the scope of preferred thermoformable materials.Thermoset resins solidify or set irreversibly when heated due tocrosslinking between the polymer chains Crosslinking can be achieved byusing nucleating agents, mold temperatures above the materials formingtemperature, radiation, etc. A thermoset resin once set or cured cannotbe softened again by heating. Thermoset resins are generallycharacterized by high thermal stability, high dimensional stability andhigh rigidity and hardness and include resins such as polyesters andurethanes.

Thermoplastic resins can be either crystalline or amorphous and can berepeatedly softened by heating. Amorphous thermoplastics includeacrylonitrile-butadienestyrene (ABS) copolymer, styrene, cellulosics andpolycarbonates. Crystalline thermoplastics include nylons, polyethylene,polypropylene and polyurethane. Examples of particularly preferredmaterials for use in the present system include thermoplasticpolyurethanes, nylons, polyesters, polyethylenes, polyamides and thelike.

In accordance with another feature of the present system, the cushioningare sealed cushions having different resistances to compression, forexample, by being filled with air, or other gas, or liquid at differentpressures, e.g., below, at, or above atmospheric pressure, or bycontrolling the number, size and/or configuration of the indentations.The indentations 126 make that part of the cushion stiffer incompression than another part of the cushion without the indentations.For example, a difference in stiffness for compression between themedial side of the shoe and the lateral side of the shoe can beachieved. Or, a smaller hemispherical radius may be used for theindentations on one side of the shoe. These variations may be used toprovide effective pronation or supination control through differences incompression between the medial and lateral sides of the shoe.

Thus, the stiffness or softness of each cushioning member iscontrollable and selectable at different areas of the heel. This featureenables the shoe to be designed with adjustable cushioning against heelimpacts during use of the footwear.

The footwear can be custom produced at the request of a customer, whocan specify the nature of the customization for one or more pairs offootwear. Each shoe of a pair of the footwear may have a differentdesign, message or message portion designed into it and rendered usingthe bed of pins described below to make the custom shoe design messagesor shapes, and then the bottom sole can be fabricated using thereformable bed described below. Once the negative is fixed in thereformable bed, suitable materials for the bottom sole can be depositedand cured and can include rubber, plastic, or foam. Furthercustomization can be done by a Computerized Numerical Control (CNC)where component design can be integrated with computer-aided design(CAD) and computer-aided manufacturing (CAM) programs. The device can beprogrammed to use a number of different tools-drills, saws, and so on.Alternatively a number of different machines can be used with anexternal controller and human or robotic operators that move thecomponent from machine to machine. Regardless, a series of steps neededto produce a part can produce a part that closely matches the originalCAD design in a highly automated fashion. In accordance with aspects ofthe subject matter disclosed herein through the use of reformable bedand a suitably programmed CNC tools, customized footwear with custom cutsole designs, can cost effectively be created in small quantities andyet scalable for mass-customization.

The sole and the cushion can be made of rubber or can be made of athermoplastic resin. Preferable materials are those which are easilythermoformable into desired flexible configurations. Materials which canbe thermoset after molding and retain the flexible characteristics forthe sole components of the present system are included within the scopeof preferred thermoformable materials. Thermoset resins solidify or setirreversibly when heated due to crosslinking between the polymer chains.Crosslinking can be achieved by using nucleating agents, moldtemperatures above the materials forming temperature, radiation, etc. Athermoset resin once set or cured cannot be softened again by heating.Thermoset resins are generally characterized by high thermal stability,high dimensional stability and high rigidity and hardness and includeresins such as polyesters and urethanes.

Thermoplastic resins can be either crystalline or amorphous and can berepeatedly softened by heating. Amorphous thermoplastics includeacrylonitrile-butadienestyrene (ABS) copolymer, styrene, cellulosics andpolycarbonates. Crystalline thermoplastics include nylons, polyethylene,polypropylene and polyurethane. Examples of particularly preferredmaterials for use in the present system include thermoplasticpolyurethanes, nylons, polyesters, polyethylenes, polyamides and thelike.

In accordance with another feature of the present system, the cushioningare sealed cushions having different resistances to compression, forexample, by being filled with air, or other gas, or liquid at differentpressures, e.g., below, at, or above atmospheric pressure, or bycontrolling the number, size and/or configuration of the indentations.The indentations 126 make that part of the cushion stiffer incompression than another part of the cushion without the indentations.For example, a difference in stiffness for compression between themedial side of the shoe and the lateral side of the shoe can beachieved. Or, a smaller hemispherical radius may be used for theindentations on one side of the shoe. These variations may be used toprovide effective pronation or supination control through differences incompression between the medial and lateral sides of the shoe.

Thus, the stiffness or softness of each cushioning member iscontrollable and selectable at different areas of the heel. This featureenables the shoe to be designed with adjustable cushioning against heelimpacts during use of the footwear.

In the context of shoe manufacturing, a computing device may be used todetermine operations of various shoe-manufacturing tools. For example, acomputing device may be used to control a part-pickup tool or a conveyorthat transfers shoe parts from one location to another. In addition, acomputing device may be used to control a part-attachment device thatattaches (e.g., welds, adheres, stitches, etc.) one shoe part to anothershoe part.

A shoe can be made from the sole or insole. An upper is fixed to theinsole by virtue of a hot-melting adhesive which is injected at the timeof assembly in the region between the lower part of the assembly insoleand a folded upper so that it is possible to assemble the upper part ofthe shoe with a single operation. Then, according to known methods, thefolded upper is glued onto the lower part of the insole and is milled soas to obtain a flat surface on which the sole is finally glued in orderto obtain the finished shoe. In the rear part, the insole generallyincludes, below a supporting base, a metal shank and a reinforcementmember which is arranged below the metal shank. The shank is generallyfixed to the reinforcement member.

The insole can have a supporting base which includes, on the lower side,a hot-melting adhesive so that said upper can adhere to said insole byheating. Preferably, the insole bears an amount of adhesive between 50and 300 g/m2 and preferably between 50 and 180 g/m2. Preferably, theadhesive has a melting point between 40 and 180 DEG C., preferablybetween 45 and 120 DEG C., when measured with the DSC method, with agradient of 10 DEG C./min. Preferably, the adhesive includes a polymerwhich is chosen among: polyesters, copolyesters, polyamides,copolyamides, polyolefins, polyurethanes, ethyl vinyl acetates, acrylicresins, polyvinyl acetates, vinyl polymers. More preferably, theadhesive is chosen among: polyesters, copolyesters, polyurethanes, ethylvinyl acetates, polyamides and copolyamides. The adhesive is formed by amix of polyurethane and polycaprolactone. Preferably, the adhesive isincluded in the lower part of the supporting base by depositing adhesivepowder. However, in the context of an extensive industrial production,it is feasible to use an adhesive in film form arranged above the lowerpart of the supporting base. The supporting base can be formed, forexample, by an impregnated fabric, by a fabric, by a non-woven fabric,by an impregnated non-woven fabric, by an extruded component, by acoextruded component, by agglomerated fibers or by reclaimed leather. Inthe case of a coextruded component, the supporting base can becoextruded together with an adhesive film, so as to automaticallyassemble the two above described parts. The reinforcement can be formedby agglomerated fibers, for example cellulose fibers impregnated with alatex. The supporting base can be provided with microperforations inorder to allow the transpiration of moisture. Thus, it is not necessaryto apply any adhesive during the assembly of the upper, considerablysimplifying this assembly step, reducing the costs and the maintenanceburdens. In particular, the complete absence of any device fortransferring a liquid adhesive at high temperature clearly drasticallysimplifies the entire assembly operation. All the downtimes that werededicated to cleaning the plant, the nozzles et cetera are furthermoreclearly eliminated entirely.

It has furthermore been noted that despite using a highly automatedmethod it is possible to obtain shoes with a flexibility that could beachieved only with old manual assembly methods but was absolutely notwithin the capabilities of currently active modern plants. Inparticular, it is possible to obtain much more flexible shoes, withobvious advantages.

Surprisingly, it is possible to use the same machines that were used forknown methods without having to perform particular modifications. Thisis a great advantage from the industrial point of view, since theimportant advantages stated above can be provided immediately withouthaving to sustain high investment costs to modify the machines requiredto assemble the shoes.

Shoe Recommendation

The foot dimensions can be used to match/best-fit the user to aparticular shoe or sole's inside dimensions. In one embodiment, theprocess can then selects footwear or a shoe with interior best matchingthe deformable/morphable foot template key sections and girths of thefootwear: (1A) Toe Section (2A) Metatarsals Section (3A) Midfoot Section(4A) Heel Section (5A) Profile of the foot (6A) plantar contour. Thebest fit shoe is shipped to the user. In one embodiment

-   -   Data Collection

Create 3D model of user's feet

Identify shoe manufacturers that best fit the user's fit

Set the best fit shoe's inside dimension with the 3D model of feet plusa predetermined gap

At point of purchase:

Run big data analysis correlating different manufacturer's sizes andcreate a correspondence between shoes from one manufacturer to othermanufacturers

Recommend best fit shoe models from the big data analysis

Another embodiment includes the following:

capturing 3D model of user's feet;

identify the subject's current best fitting shoe products;

set each best fitting shoe product's inside dimension with dimensionsfrom the 3D model plus a predetermined gap;

correlating different manufacturer's shoe sizes and creatingcorrespondences among different manufacturer shoe products; and

recommending a new shoe for the subject by looking up thecorrespondences among different manufacturer shoe products.

As part of or to assist with the size recommendation process, duringanalysis the computing device may also determine whether a user-selecteditem or shoe runs true to size. The system may make this determinationbefore or after the user selects a product by comparing the storedparameter measurements for the user-selected item to standardinformation related to standard sizes. If one or more of the measuredparameters differs by more than a threshold amount from the value(s) ofits or their corresponding reference (standard) parameters, then thesystem may determine that the model of item does not run true to size.For example, for footwear, the system may compare internal measurementssuch as length and width for the user-selected item (e.g., a size 12miming shoe) to standard internal measurements for an industry standard(i.e., a reference model) running shoe. To determine the industrystandard for a given parameter (i.e., maximum length or width of theoverall shoe interior, or of a portion such as the toe box or heel), thecomputing device may compute an overall average for the parameter forall models that are stored in the data set in that particular size(e.g., overall averages of all internal measurements of all size 12shoes). Alternatively, the industry standard size may be stored in thedata set as provided by a manufacturer, a group of manufacturers, asuppler or group of suppliers, a retailer or a group or retailers, orother similar groups. The system also may consider tactile measurementssuch as stretch or deformation and use the size corresponding tomaximum, minimum, or some intermediate level of stretch or deformationas the shoe's internal measurement in the comparison. Duringdetermination of whether an item runs true to size, the system may alsoconsider data that reflects how an item fits as opposed to, or inaddition to, determining whether the item is true to standard size. Forexample, a high heel shoe may include one or more straps that cross thetop or arch of a wearer's foot and attach at various points on the sidesof the shoe. When worn, one or more of the straps can cause the shoe tofit differently on the wearer's foot, thereby altering the fit andcomfort of the shoe. Thus, the data set for that footwear model mayinclude positive value (e.g., “yes,” “true,” or “1” for a parametertitled “horizontal straps”). The system may also consider such afeature, including various structural and decorative features integratedonto an item, that results in possible deviation from a user's selectedsize of that item when determining whether the user-selected item runstrue to size. Additionally, the system can prompt the user for secondarysizing information and receive the secondary sizing information from theuser via a user interface. Secondary sizing information may the sizethat the user is most likely to wear if their primary size does not fitproperly, or an indication of whether the user is most likely to pick alarger or smaller size if their primary size does not fit properly. Thecomputing device may use the secondary sizing information to provide amore accurate size recommendation for the user.

The system may include providing a sizing recommendation for wearableitems such as footwear based upon personalized sizing informationreceived from a user. The process accesses the wearable item data setcan include various measurements and parameters related to each of thewearable items. The computing device may receive sizing informationrelated to a specific user. For example, the sizing information mayinclude a primary footwear size and a secondary footwear size as hasbeen previously discussed. Based upon the received sizing information,the computing device may determine a personal reference size specific tothat user. For example, the user may be prompted to input their primaryfootwear size, as well as their secondary footwear sizing information.The user may input that their primary footwear size is a 12, and thattheir secondary footwear sizing information is that they typically weara size smaller than a 12 when not wearing their primary size. Based uponthis information, the computing device may calculate the personalreference size to be an adjusted size that is smaller than size 12. Theadjusted size may be a reference fraction between the user's primarysize and secondary size, such as halfway between or ⅕ of a size between.For example, for the particular user discussed above, the personalreference size may be similar to a size 11.8, or slightly smaller thanthe user's primary size. Conversely, if the user indicated that theytypically wear larger than a 12 for their secondary sizing information,the computing device may determine 406 that the user's personalreference size is similar to a size 12, or slightly larger than theirprimary size. The personal reference size need not be determined bymeasuring the user's actual foot (or other body part), but rather may bebased on internal dimensions of hypothetical (modeled) or actualreference footwear items based on data previously provided by the user,or the user's previous purchases. The computing device can determine thepersonal reference size by establishing a set of internal wearable itemmeasurements for a reference footwear model and establishing thereference size to be an extrapolated (or interpolated) size thatcorresponds to the reference model. To continue the above example, thecomputing device may determine that a user has a personal reference sizeof about 11.8. The computing device may establish a graph or othersimilar representation of all footwear that have sizing informationstored in the data set, plotting each size against each internalmeasurement for each individual piece of footwear. The computing devicemay then fit a best fit line into the data, providing a reference foreach measurement as it compares to each footwear size. The computingdevice can then locate the user's personal reference size, e.g., 11.8,on the graph for each measurement to determine a set of personalizedinternal measurements for that user. Based upon the user's personalreference, the computing device may identify a recommended size for auser-selected item, and it may provide the recommended size to the userbased on how close the user's reference size runs to an actual size,with an adjustment of the model does not run true to size. To continuethe above example, if the computing device determines that a user'spersonal reference are similar to a size 11.8 and the shoe runs true tosize, the computing device may identify a size 12 for the shoe sincethat shoe size the available size that is closest to the user'sreference size. Alternatively, for a shoe that runs larger than true tosize, the computing device may identify 408 the first available sizethat is smaller than the user's reference size. In this case, theidentified size would be 11.5. Optionally, the system may also considerstretch or deformation, and add or subtract an expected stretch ordeformation amount from the user-selected item measurements whenselecting the size of that time that is appropriate for the user.

3D Model of Body and Clothing Recommendation

FIG. 2 shows another exemplary process for creating a 3D model of thebody. The process can place the body adjacent an object with knowndimensions (40). The body can be the upper body of a person or theentire body of the person. The process then takes multiple images orvideos of the body and a reference object (42), as done above. Next,photogrammetric techniques are performed to create a 3D model of thebody (44) with dimensions based on the reference object. The processoptionally selects a standard body template and Morph/Warps the standardbody template to match 3D body model (46). Next, the process selects thebest fitting wearable item or apparel (48). The present systems includedetermining and providing sizing information to a user for a specificapparel item in response to a user selection of the apparel item from aretailer, for example, an online retailer.

As used herein, “wearable item” or “apparel” refers to any item orcollection of items that are designed, sized and/or configured to beworn by a person. Examples of wearable items or apparel includefootwear, outerwear (including, but not limited to coats, jackets,ponchos, capes, robes, cloaks, gloves, and other related outerwear),clothing (including, but not limited to, socks, pants, shorts, skirts,dresses, shirts, gowns, sweaters, hosiery, suits, underwear, lingerie,saris, wraps, swimsuits, neckwear, belts, and other related clothing),headgear (including, but not limited to, hats, helmets, glasses,sunglasses, goggles, earmuffs, scarves, and other related headgear),sporting accessories (including, but not limited to, pads, shin-guards,mouthpieces, protective sleeves, sports-specific gloves, and otherrelated sporting accessories) and other related wearable items. “Apparelmodel” or “wearable item model” refers to a specific type or version ofapparel offered by a manufacturer, typically having a name, model anditem number or code. For example, a footwear model refers to a specificmodel of footwear offered by a manufacturer. “Apparel representation”refers to a computer-readable representation of an apparel model storedin a computer readable medium. An apparel representation may be a twodimensional or a three dimensional representation. For example, afootwear representation may be a 3D representation of a specificfootwear model.

Using the deformable body models, the system can handle informal imagesof the body, for example, when standard digital camera images (e.g. cellphone cameras) are used as input and when only one, or a small number,of images of the person are available. Additionally these images may beacquired outside a controlled environment, making the camera calibrationparameters (internal properties and position and orientation in theworld) unknown.

To recover body shape from standard sensors in less constrainedenvironments and under clothing, the deformable model such as aparametric 3D model of the human body is employed. The term “body shape”means a pose independent representation that characterizes the fixedskeletal structure (e.g. length of the bones) and the distribution ofsoft tissue (muscle and fat). The phrase “parametric model” refers any3D body model where the shape and pose of the body are determined by afew parameters. A graphics model is used that is represented as atriangulated mesh (other types of explicit meshes are possible such asquadrilateral meshes as are implicit surface models such as NURBS). Thedeformable body model allows a wide range of body shapes and sizes canbe expressed by a small number of parameters. The deformable modelcaptures the statistical variability across a human population with asmaller number of parameters (e.g. fewer than 100). To represent a widevariety of human shapes with a low-dimensional model, statisticallearning is used to model the variability of body shape across apopulation (or sub-population). With a low-dimensional model, only a fewparameters need to be estimated to represent body shape. This simplifiesthe estimation problem and means that accurate measurements can beobtained even with noisy, limited or ambiguous sensor measurements.Also, because a parametric model is being fitted, the model can copewith missing data. While traditional scanners often produce 3D mesheswith holes, the deformable models can reconstruct a body shape withoutholes and without a need to densely measure locations on the body to fitthe 3D model. Only a relatively small number of fairly weak measurementsare needed to fit the model. The deformable body model also factorschanges in body shape due to identity and changes due to pose. Thismeans that changes in the articulated pose of the model do notsignificantly affect the intrinsic shape of the body. This factoringallows the combining of information about a person's body shape fromimages or sensor measurements of them in several articulated poses. Thisconcept is used to robustly estimate a consistent body shape from asmall number of images or under clothing.

In one embodiment, a method and system are described that enable therecovery of body shape even when a person is wearing clothing. Toestimate body shape under clothing, image classifiers are employed todetect regions corresponding to skin, hair or clothing. In skin regions,it is recognized that the actual body is being observed but in otherregions it is recognized that the body is obscured. In the obscuredregions, the fitting procedure is modified to take into account thatclothing or hair makes the body appear larger. The process allows forfitting the body shape to partial depth information (e.g. from atime-of-flight sensor) that is robust to clothing. Unlike a laser rangescan, most range sensors provide information about depth on only oneside of the object. Information can be gained about other views if theperson moves and multiple range images are captured. In this case onemust deal with changes in articulated pose between captures. The methodestimates a single body model consistent with all views. The disclosedmethod further uses image intensity or color information to locateputative clothed regions in the range scan and augments the matchingfunction in these regions to be robust to clothing.

In many applications it is useful to employ just one or a small numberof images or other sensor measurements in estimating body shape.Furthermore with hand-held digital camera images, information about thecamera's location in the world is typically unknown (i.e. the camera isun-calibrated). In such situations, many body shapes may explain thesame data. To deal with this, a method is described for constrainedoptimization of body shape where the recovered model is constrained tohave certain known properties such as a specific height, weight, etc. Anew method is defined for directly estimating camera calibration alongwith body shape and pose parameters. When the environment can becontrolled however, other approaches to solving for camera calibrationare possible. Additionally, a method and apparatus are described thatuses “multi-chromatic keying” to enable both camera calibration andsegmentation of an object (person) from the background.

Each body model recovered from measurements is mapped in fullcorrespondence with every other body model. This means that a vertex onthe right shoulder in one person corresponds to the same vertex onanother person's shoulder. This is unlike traditional laser orstructured light scans where the mesh topology for every person isdifferent. This formulation allows body shapes to be matched to eachother to determine how similar they are; the method makes use of this inseveral ways. Additionally, it allows several methods to extractstandard tailoring measurements, clothing sizes, gender and otherinformation from body scans. Unlike traditional methods for measuringbody meshes, the presently disclosed methods use a database of bodyshapes with known attributes (such as height, waist size, preferredclothing sizes, etc) to learn a mapping from body shape to attributes.The presently disclosed method describes both parametric andnon-parametric methods for estimating attributes from body shape.

Finally, a means for body shape matching takes a body produced from somemeasurements (tailoring measures, images, range sensor data) and returnsone or more “scores” indicating how similar it is in shape to anotherbody or database of bodies. This matching means is used to rank bodyshape similarity to, for example, reorder a display of attributesassociated with a database of bodies. Such attributes might be items forsale, information about preferred clothing sizes, images, textualinformation or advertisements. The display of these attributes presentedto a user may be ordered so that the presented items are thosecorresponding to people with bodies most similar to theirs. The matchingand ranking means can be used to make selective recommendations based onsimilar body shapes. The attributes (e.g. clothing size preference) ofpeople with similar body shapes can be aggregated to recommendattributes to a user in a form of body-shape-sensitive collaborativefiltering.

To create a codebook of deformable body models, a database of body scaninformation is obtained or generated. One suitable database of body scaninformation is known as the “Civilian American and European SurfaceAnthropometry Resource” (CAESAR) and is commercially available from SAEInternational, Warrendale, Pa. The bodies are aligned and thenstatistical learning methods are applied within the statistical learningsystem to learn a low-dimensional parametric body model that capturesthe variability in shape across people and poses. One embodiment employsthe SCAPE representation for the parametric model taught by Anguelov etal. (2005).

The system then automatically estimate the gender of a person based ontheir body scan. Two approaches for the estimation of the gender of aperson are described. The first uses a gender-neutral model of bodyshape that includes men and women. Using a large database of bodyshapes, the shape coefficients for men and women, when embedded in a lowdimensional gender-neutral subspace, become separated in verydistinctive clusters. This allows the training of gender classifiers topredict gender for newly scanned individuals based on shape parameters.A second approach fits two gender-specific models to the sensormeasurements: one for men and one for women. The model producing thelowest value of the cost function is selected as the most likely gender.In one embodiment, the process produces standard biometric or tailoringmeasurements (e.g. inseam, waist size, etc.), pre-defined sizes (e.g.shirt size, dress size, etc.) or shape categories (e.g. “athletic”,“pear shaped”, “sloped shoulders”, etc.). The estimation of theseattributes exploits a database that contains body shapes and associatedattributes and is performed using either a parametric or anon-parametric estimation technique. The gender and/or the biometric ortailoring measurements are used to select a best fitting deformable bodymodel.

Once the best fitting deformable body model is selected using the bodycodebook models, the system can match various points on the body to thedeformable model. The fitting can be performed with people wearingminimal clothing (e.g. underwear or tights) or wearing standard streetclothing. In either case, multiple body poses may be combined to improvethe shape estimate. This exploits the fact that human body shape (e.g.limb lengths, weight, etc.) is constant even though the pose of the bodymay change. In the case of a clothed subject, a clothing-insensitive(that is, robust to the presence of clothing) cost function is used asregions corresponding to the body in the frames (images or depth data)are generally larger for people in clothes and makes the shape fittingsensitive to this fact. Combining measurements from multiple poses isparticularly useful for clothed people because, in each pose, theclothing fits the body differently, providing different constraints onthe underlying shape. Additionally, the optional skin detectioncomponent within the calibration and data pre-processing system is usedto modify the cost function in non-skin regions. In these regions thebody shape does not have to match the image measurements exactly. Theclothing-insensitive fitting method provides a way of inferring whatpeople look like under clothing. The method applies to standard cameraimages and/or range data. The advantage of this is that people need notremove all their clothes to obtain a reasonable body model. Of course,the removal of bulky outer garments such as sweaters will lead toincreased accuracy. The output of this process is a fitted body modelthat is represented by a small number of shape and pose parameters. Thefitted model is provided as input to the display and clothingrecommender or an apparel fabrication system.

In addition to body shape, the match score may take into accountinformation about products such as clothing. A distance is definedbetween products. This may be implemented as a lookup table. Let pi be avector of clothing descriptors such as [Brand, Gender, Clothing_Type,Style, Size]; for example [Gap, Women, Jeans, Relaxed, 8]. The productdistance function returns the distance between any two such descriptorvectors. An exact match of brand, clothing type, style and size could beassigned a distance of zero. A match that only includes brand, clothingtype and size can be assigned a higher value. Differences in sizeproduce proportionally higher distances.

In a typical scenario, a user wishes to know if a particular garmentwith properties will fit them. A potentially similar body may have manyproduct vectors associated with it. A product distance between user andtest bodies is determined where the closest matching (minimum distance)product vector is found and this distance is returned as the overallmatch. A general product distance between two bodies can be computed asto find the two most similar product vectors for the two bodies andreturn their distance.

Additionally, stored in the database with information about products isoptional user-supplied ratings. The ratings can be used to augment theproduct match score; for example by adding a constant to it. A highrating could add zero while a low rating could add a large constant. Inthis way, both similarity of the item and its rating are combined.

When a body model is created, it may be stored in a secure database witha unique identifier associated with a user. Specifically, the shapecoefficients are stored along with the version of the shape basis used(including the date of creation and whether it was created for asub-population). This allows the body to be reconstructed, matched ormeasured independent of when it was scanned. If a pair of bodies arecreated with two different shape bases, it is straightforward (givenvertex correspondence) to convert one or both of them into a commonbasis for comparison or measurement (Section 10). Additionally,ancillary data that the user enters may be stored such as their age,ethnicity, clothing sizes, clothing preferences, etc.

A user may access their body model in one of several standard ways suchas by logging onto a website over a computer network using a uniqueidentifier and password. The body model information may also be storedon a physical device such as a phone, key fob, smart card, etc. Thisportable version allows the user to provide their information to aretailer for example using an appropriate transmission device (e.g. cardreader).

The body identifier may be provided by the user to retailers, on-linestores, or made available to friends and relatives with or withoutprivacy protection. In providing access to their body model, the usermay provide limited rights using standard digital property rightsmanagement methods. For example, they may provide access to a friend orfamily member who can then provide their information to a clothingretailer, but that person could be prohibited from viewing the bodymodel graphically. As another example, a user could provide access todisplay the body to video game software to enable the use of the modelas a video game avatar, but restrict the further transmission of themodel or its derived measurements.

When a person purchases clothing from a retailer (e.g. over theInternet) using their body model, the size and brand information may be(optionally) stored with their body model. This information may beentered manually by the user with a graphical interface or automaticallyby software that collects the retail purchase information. Optionallythe user can provide one or more ratings of the item related to its fitor other properties and these may be stored in the database inassociation with the clothing entry.

If a person has multiple body scans obtained on different dates, theymay all be maintained in the database. The most recent model can be usedby default for matching and measurement. When ancillary data is stored,it is associated with the most current scan at that time. Additionally,storing multiple body models enables several applications. For example,body measurements can be extracted and plotted as a function of time.The shape of the body can also be animated as a movie or displayed so asto show the changes in body shape over time. One method provides agraphical color coding of the body model to illustrate changes in bodyshape (e.g. due to weight loss). Since all model vertices are incorrespondence, it is easy to measure the Euclidean distance betweenvertices of different models. This distance can be assigned a color froma range of colors that signify the type of change (e.g. increase ordecrease in size as measured by vertex displacement along its surfacenormal). Color can alternatively be mapped to other shape attributes(such as curvature) computed from the mesh. The colors are then used totexture map the body model for display on a graphical device.

Collaborative filtering or recommendation uses information about manypeople to predict information about an individual who may shareattributes in common with others. A common example is movie ratings. Ifmany people who liked movie X also liked movie Y, an individual wholiked X but has not seen Y may reasonably be expected to like Y.

A new form of collaborative filtering based on 3D body shape ispresently disclosed. People with similarly shaped bodies may be expectedto be interested in similar products such as clothing or weight lossproducts. Specifically if many people with similar body shapes to X buypants of size Y, then an individual X may also be expected to fit sizeY. Thus, a body shape model is used as described to match people basedon body shape (Section 9 and 10d).

Several embodiments of this method of body shape matching are possible:

1. Size recommendation. If a user is shopping for clothing of aparticular type, the system identifies N people with similar body shapes(Section 9 and 10d) for whom ancillary data related to this (or similar)item is stored in the database (e.g. use the product distance function).A function is used (e.g. a weighed combination based on body shapedistance) to predict the best size (Section 10d). Body shape as well assimilarity in clothing preference may be used in the matching (Section9).

2. Community ratings. Instead of being presented with a specific size,the user is presented with a list of ratings for the product by peopleof similar size. The degree of similarity is shown along with optionalentries such as the rating, comments, photos, etc. The degree ofsimilarity can be expressed on a point scale or percentage scale bytaking the body shape distance measure (Section 9) and normalizing it toa new range (e.g. 1-100 where 100 is an exact match and 1 is the matchto a very different body shape).

3. Community blogs. People with similar body shapes may be trying tolose weight or increase their fitness. Shape-based matching is used tofind people with similar body shapes. Groups of people with similarshapes (an possibly preferences) define a “community”. Users can postinformation (e.g. in a blog format) about themselves and find postingsby other members of the community who of similar shape (or who haveundergone as similar change in shape). The key concept is that communityis defined based on body shape-related properties.

A seller of a particular garment can associate a body shape, or fitmodel with a garment where that body is known to fit that garment. Forexample an individual wants to sell an item of clothing that fits themthrough an on-line auction. They list the item along with a uniqueidentifier that can be used to match any other body model to theirs. Abuyer looking for clothing provides their unique body identifier and thematching component compares the 3D body shapes and ancillary data(including optional ratings of clothing fit) retrieved from a databaseto determine the match score. Given a plurality of other matches fromother fit models a display and ranking software component sorts theitems for sale based on the match score (how similar their body is tothe seller's). This method for sizing clothing applies to any retailapplication where a fit model for each clothing size is scanned and theassociated body identifier is used to determine whether a new individualwill fit that size. A score of the quality of fit (based on the bodymatch score) can be presented or a threshold on the match score can beused to identify one (or a small number of) size(s) (i.e. fit models)that will fit the user's body. This method is analogous to having afriend or personal shopper who is the buyer's size and shape and whotries on clothing for them to see if it fits before recommending it.

More generally, there may be a large database of people who have triedon the same (or similar) garment and each of them can be viewed as a fitmodel; every person in the database can be a fit model for any productassociated with them. The match distance (Section 9) between bodiesincorporates shape and other attributes. Attributes can include one ormore ratings of the product (for fit, style, value, etc.). The totalmatch score can then include a term for the fit rating indicatingwhether the garment fits the fit model. Alternatively, the match can beperformed on body shape and an aggregate fit rating for the matchedbodies computed (Section 10d). If the matched bodies have associatedreviews for the product stored in the database, these reviews may beoptionally displayed to the user such that they are optionally ranked bymatch score.

In an alternative embodiment, the match similarity is computed onlybased on product information (brand, style, size) using the ancillary orproduct distance function (Section 9). A user selects a particulargarment and a list of matches (IDs) is generated from the database whereeach ID corresponds to a person who has purchased and/or rated theproduct. The body shapes of the matching IDs are compared to the user'sbody shape by computing the body shape match score. An aggregate of allthese scores is computed; for example by computing the mean score. Thisscore is presented to the user (e.g. on a 100-point scale) to indicatehow well the garment may fit them.

Automatically Obtaining Fit for Clothing Presented on a Web Page can bedone. Using the techniques above for matching a user's body to adatabase of other bodies that have tried on similar garments, the systemincludes determining relevant clothing brand, style and size informationfrom a website. When the user clicks a button to obtain their size for agiven garment, the size determining process obtains their unique bodyidentifier. The unique identifier for the user's body model may bestored on their computer hard disk or memory, for example, in the formof a “cookie”. Alternatively, if no cookie is present, the user is askedto provide authenticating information such as a username and password.Once identified, the body shape of the user is known. The sizedetermining process searches a database for people with similar bodieswho have purchased or rated the clothing item as determined by theproduct determining process. The match score is computed and the N bestmatches are identified. The number of matches can vary but the defaultsetting in one embodiment is 10. Ratings and comments stored with the Nmatches may be displayed. Alternatively the size preferences of these Nbodies may be combined to recommend a particular size for the determinedproduct.

Measurements extracted from the body can be used as input to standardpattern generation software for custom clothing or to on-line forms forordering custom (or semi-custom) clothing.

A shape-sensitive advertising component uses the body model inconjunction with on-line (or cell phone) web browsing and shopping.Based on a person's body shape, advertising (e.g. banner ads in a webbrowser) may vary. For example, advertisers can select a range of bodyshapes that fit their product demographics (e.g. heavy men or shortwomen). The body-shape matching component matches advertiserspecifications with body shapes and presents shape-targetedadvertisements (e.g. for weight loss or plus-sized clothing). Forexample, an advertiser may specify a gender, height and weight range, abust size, etc. Advertisers may also specify body shapes based onexample 3D body models selected from an electronic presentation ofdifferent body shapes or by providing a fit model scan. These exemplarbodies are then used to produce a match score that determines howsimilar a user is to the exemplar specification.

Body shape information about a user may be stored on the user'scomputer; for example in the form of a “cookie” that provides a uniqueidentifier to an ad manager software component. The ad manager softwarecomponent retrieves information about the body from a body modeldatabase using the unique identifier. The ad manager software componentcan keep the identity of the user private and communicate generalinformation about their body shape to a shape-sensitive ad exchangesoftware component. This information may include body shapecoefficients, the ID of a similar exemplar body, measurements such asheight or weight, demographic information such as age and gender, andshape category information such as athletic or heavy build. It should beunderstood that standard ad targeting information can also be suppliedsuch as IP address, geographic location and historical click/purchaseinformation. The shape-sensitive ad exchange component matches the shapeinformation about a user to a database of advertiser requests. If thereare multiple matching advertisements, one or more of the matchingadvertisements is selected for display. The mechanism for selection canbe randomized or can take into account how much an advertiser is willingto pay. The rate for each advertisement may vary depending on theoverall quality of the match score (i.e. how close the user'smeasurements are to the target shape specified by the advertiser). Astandard bartering or auction mechanism may be used for advertisers tocompete for presentation to matched users. Statistics of purchases andadvertising-related click histories for people of particular body shapesare collected and stored in a database. Matches to the body shapes ofother shoppers or website users can also be used to target advertisingbased on the purchases of other people of similar shape. This isachieved by finding similar body shapes using the body shape matchingcomponent and accessing the stored shopping and clicking statistics forpeople of similar shape. If a person of a particular shape has clickedon an advertisement, an advertiser may pay more for presentation to asimilarly shaped person. Any website can be enabled with thisshape-sensitive advertising feature using cookies. Users can disablethis feature by changing their browser preferences. This shape featurecan be combined with other commonly acquired information about shoppingand clicking behavior used for the presentation of personalized ortargeted advertising.

The estimated body shape model can also be used to try on virtualclothing. There are several computer graphics methods, includingcommercial products, for simulating clothing draped on 3D bodies andthese are not discussed here. The body model can be saved in any one ofthe common graphics model formats and imported into a standard clothingsimulation software system.

Virtual try on is enabled by collecting a database of models ofdifferent shapes and sizes wearing a plurality of clothing items. Whenthe user wants to see how they will look in a particular clothing item,the database of stored models is searched for the closest matching bodyshape for which an image (or graphic representation) of the model inthat item exists. This image is then displayed to the user. In this way,each person visiting a retail clothing website may see the samemerchandise but on different models (models that look most like them).This provides the equivalent of a personalized clothing catalog for theperson's shape. This is a form of “example-based virtual clothing”.Rather than rendering clothing using graphics, many images of models arestored and recalled as needed. The key concept is that this recall isbased on similarity of body shape.

Virtual Make Up System

FIG. 3A shows a potential graphical user interface (GUI) with a headimage 210 that is a 2D or 3D image of the user, different types ofmakeup styluses 250 that may be picked by touch and stylus, differentwidths 240 for the makeup pencil, different lighting conditions as setforth and controlled via rulers 230 and various types and shades ofmakeup as shown in pallet 220. In operation, the user apply the makeupof his or her choice to image 220 and try various lighting conditionsand different shades until he or she is satisfied with the results.

The method of FIG. 3B includes the following:

Scan a bar code or insignia with a makeup product to retrieve colorcharacteristics of the makeup product (220)

Analyze skin pigment from 2D or 3D model of user's face and/or head(222) Receive a sequence of hand gestures and postures as well as theuse of touch and stylus forming a virtual make-up session and apply thecolor of the makeup product to user skin tone (224)

Apply virtual make-up features to the 2D or 3D model, to yield a 3Dmake-up model, based on said received hand gestures and postures (226)

Prior to taking pictures of the user, the system reminds the user tothat his/her skin is clean and free of any product, such as foundation,powder, or lotion. A sampled portion from an image can be used indetecting skin tone. Images of faces can include skin tone elements(e.g., skin of a face) and non-skin tone elements (e.g., hair, hats,sunglasses, clothing, foliage, etc.). Using a portion of an image forskin tone detection can improve the detection accuracy by maximizingskin tone elements while minimizing non-skin tone elements. The skintone detection process can extract a portion from the face image. Theskin tone detection can evaluate pixels from the extracted portion todetermine whether the pixels correspond to skin tone colors. A pixel canbe converted to one or more color spaces and analyzed to determinewhether the pixel corresponds to a skin tone. The skin tone detectionprocess can maintain information about the overall portion based on theevaluation of the individual pixels. For example, the skin tonedetection process can count the number of pixels in the portion and thenumber of pixels corresponding to a skin tone. Based on the information,the skin tone detection process can determine whether the pixels overallcorrespond to skin tone colors. For example, the skin tone detectionprocessor can compare the number of skin tone pixels to the total numberof pixels to make the determination.

The skin tone is determined by the amount of melanin, or pigment, in theuser's skin and does not change from sun exposure or skin conditions.The skin tone will be one of the following: cool, warm, or neutral. Theprocess captures the color of the veins on the inside of the wrist innatural light. If the veins appear blue or purple, the user has a coolskin tone. If the user's veins appear green, the user has a warm skintone. If the user can't tell if the user's veins are green or blue, theuser may have a neutral skin tone. If the user has an olive complexion,the user likely falls into this category.

If the user tans easily and rarely burn, the user has more melanin andthe user likely have a warm or neutral skin tone. If the user's skinburns and doesn't tan, the user have less melanin and therefore a coolerskin tone. Some women with very dark, ebony skin may not burn easily butstill have a cool skin tone. To Determine Skin Tone Step, a camera cantake a picture of a white piece of paper up to the user's face anddetermine how the user's skin looks in contrast to the white paper. Itmay appear to have a yellow cast, a blue-red or rosy cast, or it may notappear to be either, but a gray color instead. If the user's skinappears yellowish or sallow beside the white paper, the user has a warmskin tone. If the user's skin appears pink, rosy, or bluish-red, thenthe user have a cool skin tone. If the user's skin appears gray, theuser's skin probably has an olive complexion with a neutral undertone.The green from the user's complexion and the yellowish undertonecombines to create this effect. The system can recommend that the userexperiment with neutral and warm tones, since the user fall somewhere inbetween. If the user can't determine any cast of yellow, olive, or pink,the user have a neutral skin tone. Neutral tones can look good infoundations and colors on both ends of the cool/warm spectrum.

In another test, the camera takes a picture with gold and silver foil orjewelry to find the user's skin tone. For example, an image of a sheetof gold foil can be taken in front of the user's face so that itreflects light back on the user's skin. The system determines whether itmakes the user's face look grayish or washed out, or if it enhances theuser's skin. This is repeated with a sheet of silver foil and if thegold foil looks best, the user has a warm skin tone. And if thereflection from the silver foil makes the user's skin glow, the user hasa cool skin tone. If no difference (both silver and gold areflattering), then the user likely have a neutral skin tone.Alternatively, if the user doesn't have gold or silver foil, try layinggold and silver jewelry on the user's wrist, and notice which one ismore flattering.

Another embodiment analyzes images of skin behind the user's ear. If theuser have acne, rosacea, or another condition that might mask the user'sskin tone, the user can take pictures of the skin directly behind theshell of the user's ear, as this area is less likely to be affected. Thecamera can examine the skin right in the little crease behind the user'sear. If the user's skin is yellowish, then the user's skin tone is warm.If the user's skin is pink or rosy, then the user's skin tone is cool.

Next, the system uses the user's Skin Tone to Choose Colors. The systemexamines the user's skin in neutral light to find the user's complexion.The user's complexion refers to the surface shade of the user's skin,such as fair, medium, olive, tan, or dark and is not necessarily fixed.So the user's complexion may be lighter in the winter and darker duringthe summer. By looking at the skin near the user's jawline, the usershould be able to determine the user's coloring.

If the user's skin can be described as very white, pale or translucent,the user is fair skinned. The user may have freckles or a little rednessto the user's complexion. The user's skin is very sensitive to the sunand burns easily. The user may have cool or warm undertones.

If the user has pale skin that burns in the sun but then deepens into atan, the user has light skin. The user may have a little red coloringand the user's skin may be mildly sensitive and may have cool or warmundertones.

If the user tan very easily and rarely burn, the user has a medium skintone. The user likely has warm or golden undertones. If the user's skinis olive or tan year-round (even in the winter), the user's complexionis likely tan. The user almost never gets sunburn and the user'sundertone is probably neutral or warm.

If the user has warm, brown skin and black or dark brown hair, the userhas a warm complexion. The user's skin darkens very quickly in the sunand the user rarely burn. The user's undertones are almost always warm.Women of Indian or African descent often fall into this category.

If the user has very dark, even ebony skin and black or dark brown hair,the user has a deep complexion. The user may have a warm or cool skintone and the user's skin hardly ever burns.

The user's skin tone is used to choose the right colors for the user'sclothing. Matching the user's skin tone with a flattering color can helpthe user look the user's best. For example, warm undertones should tryneutrals, like beige, cream, orangey-coral, mustard, off-white, yellow,orange, brown, warm red, and yellow-greens. Cool undertones should tryblue-red, blue, purple, pink, green, plumb, navy, magenta, andblue-green. Neutral undertones can draw from both groups. Most shadeswill flatter the user's skin.

Consider the user's skin tone and complexion, the system then recommendsthe user's new lipstick. If the user has fair or light skin, try lightpink or coral, nude, beige, or dusty red. If the user has coolundertones, look for raspberry or mocha or nudes, especially. Warmundertones may want to try red with blue undertones (this will make theuser's teeth look very white, too), coral, pale pink or peachy nudes. Ifthe user has tan or medium skin, go for cherry red, rose, mauve, orberry. Deep pinks or corals will look good, too. If the user has warmundertones, focus on tangerine, orange-red, copper, or bronze. If theuser has cool undertones, look for wine colored shades or cranberry. Ifthe user have a dark or deep complexion, look for browns, purples,caramel, plumb, or wine colored lipsticks. If the user has warmundertones, try copper, bronze, or even a blue-based red. If the userhas cool undertones, look for metallic shades in ruby red or a deep wineshade.

FIGS. 4A-4H illustrate representations of representing eight of theclassic facial configurations and, as well, the “perfect” or “ideal”facial configuration of the make-up virtual face/head according to theinvention, the said make-up virtual body, face or head facialconfiguration being capable of formed selectively to any appear asselected ones of the classic facial configurations solely by means ofartistic application only of cosmetic compositions to the selected onesof the companion masks; FIG. 4A illustrating the round facialconfiguration, FIG. 4B illustrating the square facial configuration,FIG. 4C illustrating the pear-shaped facial configuration, FIG. 4Dillustrating the heart shaped facial configuration, FIG. 4E illustratingthe triangular shaped facial configuration, FIG. 4F illustrating theoblong facial configuration, FIG. 4G illustrating the diamond shapedfacial configuration, FIG. 4H illustrating the inverted-triangularfacial configuration; FIG. 4I illustrating the “perfect” or “ideal” ovalfacial configuration of the make-up virtual body, face or head headaccording to the invention;

FIGS. 5A-5E illustrate the various commonly encountered eye-formationson which the user can virtually apply eye wakeups; FIG. 5A illustratesthe typical oriental eye formation with FIG. 5A showing the method ofapplying eye shadow to such eye formation of FIG. 5A; FIG. 5Billustrates the typical mature eye formation with FIG. 5B′ showing themethod of applying eye shadow to such eye formation of FIG. 5B; FIG. 5Cillustrates the typical deep set eye formation with FIG. 5C′ showing themethod of applying eye shadow to such eye formation of FIG. 5C; FIG. 5Dillustrates the typical close-set eye formation with FIG. 5′ showing themethod of applying eye shadow to such eye formation of FIG. 5D; FIG. 5Eillustrates the basic bulging eye formation with FIG. 5E′ showing theapplication of eye shadow to the eye formation of FIG. 5E.

Directing attention to FIGS. 5A to 5E, five sets of commonly encounteredeye formations are automatically detected by the computer and eye makeupis recommended. Upon acceptance of an eye make up pattern and aparticular product, the product's color and texture information isretrieved and rendered accurately with the user's skin tone. FIGS. 5A-5Eare illustrated along with relatively matching illustrations FIGS.5A′-5E′; showing one eye, the right eye as viewed, of the eye formationsrespectively of said eye formations 5A-5E to illustrate the mannertaught to the student or trainee how to utilize the three conventionaltypes of eye-shadow in treating the respective eye to accentuate thoseshapes. There are five different eyeshapes illustrated, namely, theoriental shape shown in FIG. 5A, the natural mature wide set eyeformation shown in FIG. 5B, the deep set eye formation shown in FIG. 5C,the closed set eye formation shown in FIG. 5D and the basic bulging eyeformation shown in FIG. 5E. The eye formation carried by the companionmask members can be utilized to teach and practice the use of differentshading compositions selected and employed to illustrate the applicationof eye shadow to alter or to reinforce the use of an eye shadowcomposition to selectively emphasize the training and practice ofapplying eye shadow cosmetic make-up of the three conventional shadesrespectively to the masks, to enable the trainee to treat the variousdifferent eye formations encountered. These types of eye shadow comprisethe range, light eye shadow 76, medium dark eye shadow 78 and very darkeye shadow 80.

FIGS. 5A′-5E′ illustrate the method of applying three different shadesof eye shadow to the eye formations of FIGS. 15A-15E, only one eye, theright eye as viewed, being shown in said FIGS. 15A′-15E′.

FIG. 5E′ presents the oriental eye formation of FIG. 5A for which thelight eye shadow 76 is applied at a location across the upper eye lidextending close to the bridge 41 of the nose 42. Then, the very dark eyeshadow 78 is applied along a line extending across the eye lid extendingalong the eye lid in a line following the eye socket. The remainder ofthe eye lid receives an application of medium dark eye shadow 80thereacross.

FIG. 5B′ presents of the natural mature natural wide-set eye formationfor which the light eye shadow 76 is applied to the eye lid along thearea adjacent to the eye brow with dark eye shadow 78 being applied toright corner of the illustrated eye lid while the medium density eyeshadow 80 is applied to the remainder of the eye lid of the companionmask 14A to result in the treatment applied to the companion mask 12 togive training to the trainee so as to result in the mature appearance.

In FIG. 5C′, the deep-set eye formation of FIG. 5C, is shown as thedesired result of application of the shades of eye shadow to the eyeformation of the companion virtual body, face or head mask to result inthe deep-set appearance. Instead of using a substantial coverage of theeye lid, the medium density eye shadow 80 is applied to the upper rightcorner of the eye lid along the area thereof closely adjacent the rightinner portion of the eye lid including a portion near the bridge 41 ofthe nose 42. Dark eye shadow 78 is applied to the right corner of theeye lid and along the area at the bridge 41 of the nose 42. Theremainder of the eye lid is treated with light eye shadow 76.

FIG. 5D′ illustrates the close-set eye formation of FIG. 5D on whichmedium eye shadow 80 is applied at the upper portion of the eyeformation 15D′, the light shadow 76 and the dark density eye shadow 78is applied upon the inner half of the eye lid to reach the right cornerof said eye lid closely adjacent the bridge 41 of the nose 42. Theremaining half of the eye lid receives the light shadow to the left areaof the eye lid to result in the close set eye formation.

FIG. 5E illustrates the basic bulging eye formation which receives theapplication of light eye shadow 76 along the upper area of the eyesocket to the upper right hand corner of the right hand portion of theeye lid extending to the right side of the bridge of the nose. The darkeye shadow 78 is applied below the area occupied by the light eye shadow76 and extends from the left corner of the bridge of the nose below thelight eye shadow partially along the left eye socket from the bridge ofthe nose. The remainder of the eye lid receives coverage of a mediumdensity shadow 80.

FIG. 6A illustrates a virtual application of cosmetic preparations suchas light foundation creme, eye disguise and/or highlighter to the faceimage or 3D model so as to effect an improvement of the facialappearance, reducing the effect of various encountered areas on thefacial configuration which are improved. The face model or image 14 isillustrated having various areas of improvement to which the applicationof light foundation creme, eye disguise, highlighter, concealer or blushwill result in a change in the light reflection angle so as to changethe visible impression to the viewer. In this way, an illusion iscreated reducing the viewer's recognition of the undesired feature.Application of shadow at the cheek areas 81 of the mask as shown in FIG.6A will cause the viewer to see the visage or facial configuration asthinner than its physical reality. Likewise, a thick or wide nose can bemade to appear thinner to the viewer by application of a lightreflective foundation creme to the sides of the nose bridge at area 82of the mask as shown in FIG. 6A. The visual effect of deep-set eyes canbe disguised by application of light foundation or eye disguise to thearea 83 at the bridge of the nose between the inner right hand corner ofthe right eye as shown in FIG. 6A to the degree that the reflection oflight is changed. The appearance of dark circles under the eyes can beameliorated by applying concealer instead of highlighter at area 84 ofthe mask as shown in FIG. 6A. Light foundation can be applied at area 86of the companion mask 16 as shown in FIG. 6A which application changesthe reflection of light thereat to reduce the visible appearance of areceding chin. The appearance of a low forehead can be modified byapplication of light foundaton creme at area 88 of the companion mask 14as shown in FIG. 6A. Area 94 of the companion mask 14 illustrated inFIG. 6A is receptive of application of light foundation creme to reducethe visual effect of a long face.

FIG. 6B illustrates the areas of the mounted mask to which darkfoundation or eye disguise can be applied to conceal or reduce theappearance of certain perhaps objectionable characteristics of thefacial configuration. The appearance of a long nose can be disguised byapplication of dark foundation or shadow along both sides of the nose atareas 90 and 92 of the facial configuration of the mask 14 as shown inFIG. 6B. The appearance of a protruding forehead can be disguised byapplying dark foundation to the area 98 of the mask 14 as shown in FIG.6B. If the undesired appearance of a large nose is to be reduced,application of dark foundation at area 100 of the companion mask shownin FIG. 6B effects contouring the center of the nose. The trainee can betaught and can practice the techniques to disguise or conceal theappearance of a wide nose by contouring along both sides of the nose atareas 90 and 91 of the companion mask 14 shown in FIG. 6B. A hooked nosecan be disguised by contouring the protruding bone by contouring thearea 102 of the protruding bone of the companion mask 14 shown in FIG.6B. A crooked nose can be made to appear straight by contouring thecrooked side along area 106 with dark foundation and highlighting thecenter line 108, stopping at the bulb 110 at the end of the nose of thecompanion mask 14 as shown in FIG. 6B. The appearance of a double chincan be corrected by contouring the entire length of the chin at area 112of the companion mask 14 with dark foundation or eye disguise while theappearance of a long chin can be improved by applying dark foundation bycontouring the area 114 at the center only of the chin of the companionmask 14 as shown in FIG. 6B. Likewise, a square prominent chin can bereduced in appearance in the same manner as the double chin isminimized. The appearance of large jaws can be reduced by applying canbe concealed by application of dark foundation or eye disguise alongarea 116 at the upper cheek of the companion mask in the vicinitybetween the right eye and the lower portion of the ear in FIG. 6B. Awide nose requires application of dark foundation creme or eye disguisecontouring both sides of the nose at areas 118 as shown in FIG. 6B. Theappearance of heavily lidded eyes can be ameliorated by application ofdark foundation or eye disguise at area 120 of the companion mask shownin FIG. 6B. Area 122 of the companion mask 14 illustrated in FIG. 6A isreceptive of application of light foundation creme to reduce the visualeffect of a long face. A long nose requires application of darkfoundation at areas 122 as shown in FIG. 6B contouring the base of thenose and the tip thereof as carried by the companion mask 14.

The above discussion is intended to illustrate the operation of themasks 14 of the make-up virtual body, face or head masks of theinvention as intended for use by trainees or students of cosmetology inthe course of their studies in beauty schools and the like. As mentionedearlier, the trainee or student user of the make-up virtual body, faceor head kit has, in a portable carrier, the full complement of make-upvirtual body, face or head, masks of selective different color,materials and tools, providing portability enabling the trainee orstudent to learn and practice both in the school and at home or otherlocation outside the school area. Not only do the trainee or studenthave access to the required tools anywhere, but also, with the soft-skinmake-up virtual body, face or head and plural soft-skin masks of thevaried different ethnic colors to select, the soft-skin virtual body,face or head enables the trainee, student or other user of the kit tolearn and practice the art of facial massage due to the softness andflexibility of the surface and texture of the material.

In FIGS. 7A-7I, there are shown plural representations of likely lipformations—than is lip line formations—representative of the lip linesto be applied by the art of permanent make-up, that is by tattootechnique. In the figures set forth, FIG. 7A illustrates a lip formationwhere both upper and lower lips are thin; FIG. 7B illustrates a lip lineformation comprising a thin upper lip compared to a thick lower lip;FIG. 7C illustrates a lip line formation comprising a relatively thinupper lip line is combined with a thicker lower lip line. FIG. 7Dillustrates a small line mouth with the lower lip line is thicker thanthe upper lip line. FIG. 7E illustrates a lip formation having an arrowbow configured upper lip line and the lower lip is thicker than theupper lip. However, the lip line of the pair shown in this FIGURE droopsat its ends, particularly the lower lip line droops. FIG. 7F provides anoutline of an eye formation wherein the lip outlines comprise a largefull centered pair of lips with tight corners where as FIG. 7Gillustrates lip outlines which illustrates a small full centered pair ofupper and lower lip outlines. FIG. 7H illustrates lip outlinesrepresenting a small, uneven pair of upper and lower lips, the upper lipoutline being arrow bowed and the lower lip outline being substantiallythicker than the upper lip outline. FIG. 7I illustrates a lip lineformation where the upper lip is thin and uneven as well as arrow bowed,the lower lip outline also is thicker than the lower lip outline but isnot as sharply bowed as the lower lip outline shown in FIG. 7H.

Clothing Fitting System

A user can be a consumer of fashion items. Further, a user may includean expert such as a fashion professional, where this fashionprofessional includes a fashion designer, a personal shopper, a personalstylist, a journalist who reports on fashion, or some other suitableperson. Matching is based, in part, upon an attribute of a fashion item.Attributes include color, fabric, cut, designer, size, time of creation,and other suitable attributes used in denoting a fashion. In one exampleembodiment, matching includes receiving input in the form of aparticular fashion item, and based upon this fashion item suggesting anadditional fashion item based upon matching attributes. Matching isfacilitated through the use of a learning machine. In one exampleembodiment, the learning machine can determine the user's favoriteactress or model and build a related association. Thus, the fashionstyle from trendy stars, models, and musicians followed by the user insocial media (facebook, twitter) can be analyzed and similar items canbe added to the virtual digital closet of the user. Similar clothingitems worn by the actress or model can be retrieved and the clothingdata can be superimposed on the 3D model of the user to show the userthe expected appearance. In addition, experts such as magazineeditors/writers can provide outfit ideas and the user can apply theoutfit ideas to his/her wardrobe and hairstyling such as suggestions athttp://www.glamour.com/fashion/outfit-ideas, among others. The user canadjust color or size or any other attributes of the clothing, and placean order. The order is sent to the fashion maker, who customizes theitem accordingly and ships the product to the user.

One exemplary process is as follows:

-   -   Identify user's current fashion style. The fashion item can be        hair styles, clothing, shoes, bras, dental wear        (braces/aligners)    -   Determine matching style sets based on one or matching        attributes, current fashion style, fashion experts, or celebrity        styles    -   Present matching style sets and get user selection    -   Identify vendor of the user selected style    -   Render a 3D augmented reality imposed over the user 3D model for        preview    -   Accept user customization requests and re-render the 3D        augmented reality view    -   Send data to the vendor to ship fashion product to the user

FIG. 8A shows an exemplary process to provide mass-customized clothing

Capture 3D model of clothed body (310)

Digitally remove current dress (312)

Select new fashion styles from new trends (314)

Morph or project clothing onto the 3D model of body (316)

Allow user to iterative change fashion color, length until satisfiedwith new clothing (318)

Allow user to select from a library of jewelry and shoes to providerealistic simulation (320)

Order desired clothing with custom measurements (322)

In one embodiment, trendy clothing identified based on news or Internetbuzz can be located. In this embodiment, the system performs a search ofan inventory of fashion items based on the identified fashion preferenceof the user and then performs a similarity search of the fashioninventory and generates similarity search results (e.g., fashion itemsthat are similar to an item previously worn by the user).

In some example embodiments, the similarity search is based on an imagesearch of the style or the identification of similar clothing. In someinstances, the similarity search is also based on one or moremeasurements of the user's body. For example, to perform a similaritysearch, the search module compares one or more visual features extractedfrom the fashion image and one or more visual features identified in animage depicting a fashion item included in the fashion inventory beingsearched. In some example embodiments, the similarity search is based onan attribute-value pair that characterizes a fashion item previouslyworn by a celebrity, model, star, user's friend, user's social networkor even by the user. In some instances, the similarity search is alsobased on one or more measurements of the user's body. The performing ofthe similarity search may include selecting an attribute-value pair tobe used as a query attribute-value pair and identifying one or moreitems in the fashion inventory that are characterized by theattribute-value pair. In some instances, the coordination search is alsobased on one or more measurements of the user's body. The performing ofthe coordination search may include identifying, based on a coordinationrule, one or more items in the fashion inventory that can be worntogether with the fashion item that corresponds to the query image.Similarly, the search module may perform a similarity search based onthe first image and then may generate one or more similarity searchresults. The search module may perform a coordination search based onthe second image and further generate one or more coordination searchresults.

As still yet another embodiment, a feature to perform similaritysearching is a user's geographical location used as a criterion to matchthe user to other users as style may vary from one location to anotherlocation. To illustrate this type of feature by way of example only isthe case where a user A is a business executive woman living in CountryA1. She is travelling on a business trip to country B1. Country A1 couldbe a conservative country, where ethnic attire is normally worn even inbusiness meetings. Country B1 could be a fashion conscious country. UserA may want to wear fashionable clothing, that would look smart on herand be appropriate for business meetings in B1.” In yet another exampleof using this type of feature of engine 110 is illustrated as follows, auser is a graduating 22 year old female, 5′3″, 145 lbs, having ahourglass body shape, starting a new job as a project manager in NewYork. She does not know what kind of attire is appropriate and whatstyle of clothing would look good on her. She creates an account on thesystem of the present invention, creates her profile, enter herattributes, instantly gets matched to other users of similar attributesand gets recommendation on what style of attire would look good on her.In another preferred embodiment, the system may her to purchase the itemin her size.

Based on the above fashion recommendation, the system can fabricatemass-customized clothing for the user that is trendy. The process has atraining phase and a run-time phase as follows:

Training

Collect library of fashion designs (Pinterest, Google image, facebookimages)

Normalize the fashion designs to standard size

Break up the designs into sections and extract features of sections

Clusterize the features into a codebook of clothing designsections/elements

Train library and create probabilistic model (HMM) to represent afashion design as a collection of sections

Run-Time

Generate 3D model of user

Receive images of target fashion worn by celebrity or model for the user

Normalize target fashion images to a standard size

Extract features from the target fashion images

Apply probabilistic model to features and create a 3D model of thetarget fashion model

Do virtual fit to the user 3D model and optimize the 3D clothing modelfor crotch fitting and movement stress

Flatten the 3D clothing model into 2D pattern

Present to laser cutter to cut pattern

Move cut pattern to computerized sewing system

Perform QA as needed and ship mass-customized clothing to user

In one embodiment, the system receives images of trendy clothing orclothing style from a celebrity that the user wishes to match. Theimages can be extracted from various news outlet or celebrity sites, forexample from google images with search “celebrity fashion” or “tomcruise fashion”, for example. For both training and run-time, the systemgenerates features associated with portions of the fashion clothing. Inone embodiment, a L-C-S-H process can be used, forlow-level-characteristics-selection mask-high level. Low-level refers tothe features that are calculated from the images of each article (e.g.,2D shape, 2D color, 2D texture, 2D edges, and 3D shape). Characteristicsrefer to the attributes that are found on generic articles of clothing(e.g., pockets, collars, hems). Selection mask refers to a vector thatdescribes what characteristics are most useful for each category (e.g.,collars and buttons are most useful for classifying shirts), used as afilter. High level refers to the categories to be classify (e.g.,shirts, socks, dresses). In one embodiment, the unique characteristicsto differentiate categories of clothing can be: Collar, Top brackets,Dark colored, Denim, Ankle hem, Front pockets, Graphic pictures,Colored, Plaid, Thigh hem, Back pockets, Graphic texts, White colored,Patterns, Inseam, Side pockets, Belt loops, V-neck, Round neck, Elasticband, Top buttons, Striped, Bicep hem, Front zipper, Shoulder hem, Wristhem, or Shin hem, among others.

The L component of the approach uses the low-level features to estimateif the article does or does not have a particular characteristic. Thelow-level features that were used in this approach consist of colorhistogram (CH), histogram of line lengths (HLL), table point featurehistogram (TPFH), boundary, scale-invariant feature transform (SIFT),and fast point feature histogram (FPFH). To combine the low-levelfeatures of all five instances into a single value or histogram, eachvalue is determined by averaging each individual value along with itsneighbors, in the case of the histogram.

For the part of the algorithm that converts from low-level tocharacteristics, low-level features are compared to the variouscharacteristics. Since the characteristics were binary values, libSVM isused to solve the two-class problem. Each low-level feature determinesif the characteristic is in class 1 or 2. Class 1 contains positiveinstances and class 2 contains negative instances.

For an article of clothing, a high definition RGB image and a raw depthmap and background subtraction is done on the RGB image to yield animage of only the object. The background subtraction is performed usinggraph-based segmentation. Once the object is isolated within the image,multiple features are calculated from the RGB image and the 3D pointcloud. These features capture 2D shape, 2D color, 2D texture, 2D edges,and 3D shape for both global and local regions of the object. Oneimplementation uses Felzenswalb and Huttenlocher's graph-basedsegmentation algorithm which uses a variation of Kruskal's minimumspanning tree algorithm to iteratively cluster pixels in decreasingorder of their similarity in appearance. An adaptive estimate of theinternal similarity of the clusters is used to determine whether tocontinue clustering.

A color histogram CH is a representation of the distribution of thecolors in a region of an image, derived by counting the number of pixelswith a given set of color values. CH are chosen in this work becausethey are invariant to translation and rotation about the viewing axis,and for most objects they remain stable despite changes in viewingdirection, scale, and 3D rotation. CH is used to distinguish, forexample, between lights and darks, as well as denim.

A Table Point Feature Histogram (TPFH) feature consists of a263-dimension array of float values that result from three 45-valuesubdivisions, that are calculated from extended fast point featurehistograms (eFPFH), and 128-value subdivision for table angleinformation. This feature is a variant on the viewpoint featurehistogram. The eFPFH values are calculated by taking the difference ofthe estimated normals of each point and the estimated normal of theobjects centerpoint. The estimated normals of each point and thecenterpoint are calculated by projecting them on the XY, YZ and XZplane.

A boundary feature captures 2D shape information by storing theEuclidean distances from the centroid of each article to the boundary.First, the centroid of each binary image is calculated containing theobject (after background subtraction). Then, starting at the angle ofthe major axis found by principle components analysis, 16 angles thatrange from 0 to 360 (i.e., 0 to 337.5) are calculated around the object.For each angle, the process measures the distance from the centroid tothe furthest boundary pixel

Other feature includes histogram of line lengths (HLL) to helpdistinguish between stripes, patterns, plaid, and so forth. For this, weuse the object image as before (after background subtraction) andcompute the Canny edges, then erode with a structuring element of onesto remove effects of the object boundary.

Next, local features can be computed. The SIFT, scale invariant featuretransform, descriptor is used to gather useful 2D local textureinformation. The SIFT descriptor locates points on the article (afterbackground subtraction) that provide local extremum when convolved witha Gaussian function. These points are then used to calculate a histogramof gradients (HoG) from the neighboring pixels. The descriptor consistsof a 128-value feature vector that is scale and rotation invariant.

A FPFH, fast point feature histogram descriptor can be used to gatherlocal 3D shape information. The FPFH descriptor utilizes the 3D pointcloud and background subtraction for each article and segments thearticle from the background of the point cloud. For each 3D point, asimple point feature histogram (SPFH) is calculated by taking thedifference of the normals between the current point and its neighboringpoints with a radius. Once all of the SPFHs are computed, the FPFHdescriptor of each point is found by adding the SPFH of that point alongwith a weighted sum of the neighbors. Other features and descriptors canbe used.

Once the features are computed, the global features are concatenated tocreate a histogram of values. For local features, SIFT and FPFH arecalculated separately through bag-of-words to get two element histogramsof codewords. Being concatenated, this yields predetermined values forthe local features. Then being concatenated with global features yieldsadditional values, which are then fed to the multiple one-versus-allSVMs.

With the codebook, one embodiment can train a probabilistic learningsystem such as Hidden Markov Model (HMM) to represent a fashion designas a constrained combination of various esthetic components or sections.HMM is a statistical Markov model in which the system being modeled isassumed to be a Markov process with unobserved (hidden) states. A HMMcan be presented as the simplest dynamic Bayesian network. In simplerMarkov models (like a Markov chain), the state is directly visible tothe observer, and therefore the state transition probabilities are theonly parameters. In a hidden Markov model, the state is not directlyvisible, but the output, dependent on the state, is visible. Each statehas a probability distribution over the possible output tokens.Therefore the sequence of tokens generated by an HMM gives someinformation about the sequence of states. The adjective ‘hidden’ refersto the state sequence through which the model passes, not to theparameters of the model; the model is still referred to as a ‘hidden’Markov model even if these parameters are known exactly. A hidden Markovmodel can be considered a generalization of a mixture model where thehidden variables (or latent variables), which control the mixturecomponent to be selected for each observation, are related through aMarkov process rather than independent of each other. Recently, hiddenMarkov models have been generalized to pairwise Markov models andtriplet Markov models which allow consideration of more complex datastructures and the modelling of nonstationary data. Other machinelearning/classifiers can be used, as detailed below.

Once the system is set up, it can recognize and recreate clothing givenan input image. For example, if a celebrity wears a fashionable clothingto attend a highly publicized event, images of the celebrity arepublicized and the user may want to replicate the look. In this case,the system retrieves a 3D model of the user and receives images oftarget fashion worn by celebrity or model liked by the user. The systemnormalizes the target fashion images to a standard size and extractsfeatures from the target fashion images as detailed above. The systemapplies the probabilistic model to features and creates a 3D model ofthe target fashion model using the HMM recognizer using thecomponents/sections model. The system can then apply a virtual fit tothe user 3D model and optimize the 3D clothing model for crotch fittingand movement stress.

In one embodiment, since the optimization problem is combinatorial andthe number of combination items can vary (e.g., a pockets can be addedor removed), it is difficult to define a closed-form solution. In fact,as in the real world, it is desirable to obtain multiple optimalsolutions (outfits) from the various components instead of a singleglobal optimum. The process generates candidate solutions by sampling adensity function defined over the space of possible outfits. The densityfunction is defined using idealized analytical formulations. Sampling ispreferably performed using a Markov Chain Monte Carlo sampler. Duringthe optimization dimensionality may change; i.e., the number of clothingcomponents/sections may be altered during the optimization process, anda Reversible Jump MCMC (RJMCMC) framework supplements parameter-changingdiffusion moves of Metropolis-Hastings (MH) with an additional set ofdimension-altering jump moves, which allow the chain to move betweensubspaces of different dimension. To efficiently explore the solutionspace, a simulated annealing technique is applied in the optimizationprocess with a Boltzmann-like objective function. A dimension matchingstrategy is adopted to allow reversible jumps across subspaces ofdifferent dimension or within the same subspace. The RJMCMC can be usedto define the jump moves as adding/removing a clothing item to/from theoutfit, which induce a dimension change, and diffusion moves as swappingitems or modifying an item's colour, which involve no dimension change.

Various fitness criteria can be applied to evaluate how well the newdesign fits the user, such as comparing lengths and areas, analysis ofspace between clothing and the body, among others.

The final clothing model is flattened into 2D pattern. In oneembodiment, each mesh element is deformed during the flattening from 3Dto 2D plane. A pattern projection can be used that takes intoconsideration elastic and shear properties of the fabric.

One embodiment uses Design Concept Tex Tech (DCTT) software. Theflattening operation involves two steps: first, the selection of theregion part to be flattened, and second, the selection of flatteningoptions appears in the “Flattening parameters” dialog box. Theflattening tool provided by DCTT considers the geometric constraint ofthe shape, but no material properties. The process is comparable to theflattening of a network of springs whose stability is obtained throughan even distribution of its internal energy. It is an iterative processwhich proceeds layer by layer beginning from the flattening start point.To control the flattening process, DCTT offers both automatic andnumerical options, as can be seen in FIG. 3-6. The automatic controlallows the flattening algorithm to run until balance is reached. Thisoption is always selected for pattern flattening throughout thisresearch work. To prepare the flattened pattern pieces for meaningfuluse in clothing manufacturing, an appropriate seam allowance is addedaround them. This is done in a “2D product” document using the “designparts” tool with the “create seamline part” option, which is availableunder the “parts” menu. Rendering of the virtual clothing itemsdeveloped on the 3D templates is performed by keeping both the “3DDesign” document (containing the 3D template and virtual clothingdesign) and the “2D pattern” document (containing the flattened patternpieces open). On the active “2D Pattern” document, the “createrendering” tool with a “create virtual marker” option available underthe “Pattern” menu is used to apply different graphical images of aparticular pattern piece. After doing that, when the “realisticrendering” option in the “rendering” tab is activated, the graphic imageapplied on the pattern pieces is visualized in 3D.

In another embodiment, from the 3D model of the user, 2D non-contactanthropometric and automatic pattern generation system of men's shirtsand pants can be done. The system needs the users to provide front andside photos of the target celebrity or model, and carry interactiondesign of shirts or pants' styles. Then the patterns can beautomatically generated based on the user's physical measurement, theceleb photos and the style design. The two-dimensional anthropometricsystem is integrated with automatic pattern generation system. In oneexample, for men's trousers, an automatic pattern generation can applypredetermined rules on the 3D model of the user such as those by Hong Xuin Pattern Automatic Generation for Men's Trousers, ISSN: 2005-4297 IJCA2014 SERSC.

The 2D pattern can be sent to a cutter such as a CNC or a laser cutterto cut patterns from a fabric that is selected by the user, and thesystem can move the fabric with the cut pattern on a conveyor to acomputerized sewing system, where a robot can pick up the pattern andthe sewing system can assemble the pieces into mass customized clothing.The machine fabrication can extend into computer controlled weaving,dying to create a highly mass-customized fashion wear for users, asshown in FIG. 8C.

In some embodiments, the user's purchase history, retained as part oftheir account information, may be used as training data for a Bayesiannetwork component associated with the outfit suggestion component. Forexample, if the user has previously purchased high heel shoes and ajacket in the same purchase, the Bayesian network component mayrecognize that sports shoes and vests should both belong to the outfit.

The present 3D clothing design systems offer a number of benefits over2D clothing design systems in use. Virtual prototyping usingcomputer-based 3D clothing product-development techniques results infewer physical prototypes and a shorter product-development phase. Fordecision-making on product selection and prior to the commencement ofproduction, it is usual for at least two up to ten physical prototypesto be made when using existing traditional product-development systemsand this incurs a high cost involvement and time consumption. Virtualprototyping and virtual try-on processes can significantly reduce theproduct-development time and cost. Virtual review and evaluation of fitwith realistically simulated fabric behaviour can enable fasterdetection of errors and earlier corrections to design elements, materialselection and assembly. At the same time, the virtual prototypes can beused as a marketing aid for online product presentation andinternet-based retailing. The application of flattening technologyprovides the opportunity to combine clothing design and pattern creationin to a single step. Automatic flat pattern extraction from 3D designsoffers a considerable reduction of the time and manpower involvement inthe pattern cutting process. The instant 3D CAD system will form thenerve centre of at the centre of a textile information network.

Crowd Based Feedback for Manufacturers

The process includes collecting, aggregating and analyzing the feedbackinformation, according to an embodiment. For example, the recommendationsystem may receive an inquiry for a user-selected wearable item andgenerate a recommended size for the wearable item for the user. Morespecifically, the recommendation system may access a database includinga plurality of representative model of wearable items. Therecommendation system may also access a user's profile to determine asize of a wearable item the user has indicated they have previouslyowned or worn. Based upon a comparison of the profile data and therepresentative models of the user-selected item, the recommendationsystem can generate and present the size recommendation to the user. Theuser can then opt to purchase the recommended item, and therecommendation system (or a purchasing computer or system associatedwith the recommendation system) can complete the transaction and updatethe user's profile to indicate that the user has purchased the item.After the user has purchased the item, the recommendation system maygenerate and/or send a message to the user to prompt the user to providefeedback regarding the recommended size for the purchase item. Forexample, an email or other similar electronic message may be send to theuser after a period of time has elapsed from the purchase date.Alternatively or additionally, the user may be prompted to providefeedback information the next time that they access the recommendationsystem. The feedback may be an assessment of the fit and/or performanceof the item as described above. Analysis of the feedback information mayinclude analyzing the individual user's feedback for any anomalies orinformation that would indicate an error by either the user or therecommendation system. Similarly, the feedback information may beanalyzed to determine that the user received the correct product. Forexample, if the user indicates that the overall length of the item theyreceived is off by more than an acceptable amount, it may be determinedthat the user has received an improperly marked or manufactured item.

Additionally, the user feedback can be combined with additional userfeedback related to the same item to provide a group analysis of boththe recommendation system's output for that item (e.g., how accurate isthe recommended size being output for that item) as well as to determineany trends related to the manufacturer of that item (e.g., nearly 30% ofall users report that the item runs much smaller than the size wouldindicate). Such a group analysis can provide a larger scale view of boththe recommendation system's recommendation as well as the manufacturingcharacteristics of the item. Based upon the analysis, the recommendationsystem can use the feedback information to improve the recommendationsystem. For example, if users are consistently indicating that a sizingrecommendation for a particular shoe is wrong, and that the actual sizeof the shoe is smaller than recommended, the recommendation system mayrecognize that a high number of users are leaving negative feedback, andprovide a report or an indication to an administrator or other similarpersonnel that the stored measurements for that particular item may needto be reviewed. Thus, the system may use the feedback to determinewhether or not a particular footwear model runs true to size. Therecommendation system can also use the feedback information to identifyproducts that receive at least a threshold amount of positive feedbackor negative feedback, as well as trends among products. For example, therecommendation system may identify a product where a high percentage(e.g., over 90%) of purchasers are providing positive reviews. Therecommendation system may then be more likely to provide that item as arecommended item for purchase based upon the historically positivefeedback. Additionally, user feedback can be used to evaluate newrecommendation algorithms, and determine which, if any, aspects of thenew recommendation should be maintained or eliminated. For example, therecommendation system may adjust the recommendation algorithm to place ahigher or lower weight on certain fit aspects than others whengenerating a recommendation. However, if the feedback related torecommendations using the new algorithm are generally negative, therecommendation system may automatically tweak or otherwise alter the newalgorithm to change which fit aspects are more highly weighted.Similarly, a system administrator or other software programmer workingwith the recommendation system may tweak or otherwise alter the newalgorithm. Conversely, if the feedback related to recommendations usingthe new algorithm are generally positive, aspects from the new algorithmmay be incorporated into existing algorithms as well. Similarly, therecommendation system can be used to provide suppliers or manufacturersof the items being reviewed with the feedback information. Therecommendation system may monitor the feedback information to identifyone or more trends in the information such as a collection of reviewershaving the same or similar negative feedback regarding an item. If thenumber of reviewers exceeds a particular threshold as set by themanufacturer (e.g., 25%), the recommendation system may be configured toprovide the manufacturer with a notice indicating the negative feedback.In addition to merely providing an indication of feedback, the noticesto the manufacturer may include a recommendation that the manufactureralter one or more physical components of the item, adjust an internal orexternal dimension of the item, change a material used in themanufacture of an item, or manufacture a new item that combines severalliked features (or eliminated several disliked features) from one ormore reviewed items. Additionally, feedback received from a particularuser can be used to develop a customized product specifically for thatuser. The system may provide a user's individual feedback to amanufacturer, and the manufacturer may contact the user to inquire aboutcreate a customized product specifically for that user. For example, auser may indicate that nearly all fit aspects of a particular shoe arehighly rated, but that the overall width of the toe box is too tight.The manufacturer of the shoe may receive the feedback, and contact theuser with alternative footwear that may better suit their sizingrequirements, or with the option to create a customized product. Forexample, professional athletes or other similar consumers with a highdemand for proper fit, may use the recommendation system and feedbackcollection mechanism as described herein to work with a manufacturer toproduce a properly fitting article of clothing.

Hairstyling Recommendation

In another embodiment, the system can analyze fashion trends based onthe user's social network profile, the twitter follow profile, celebritylikes and followings, or expert advice from his/her fashion advisor orhairdresser or experts from magazines and a number of sources. Asanother alternative or addition, another feature that may be used bymirror to perform similarity searching may correspond to a user's heightrange to height range; weight-weight, body shape-body shape; age groupappropriate as explained above; profession range-profession range; jobposition-job position; geographic location to geographic location anduser profile attributes to celebrity profile attributes. For instance,the height and the width of a user's body figure, as well as ageneralization or quantitative description of an overall head-shape(elongated or round shape) may provide another basis for identifyingresults. Additionally, a user may perform some actions on other user'simages. For example: If the user likes the way clothing fits on anotheruser, ii) likes the style of clothing, iii) likes clothing by brand, iv)type of clothing or for any other reason, the user can “like” the image,leave a “comment” on the image, or save the image in his/her digitalcloset.

The system first captures the user's head 3D model. For stylerecommendation, the hair color information, the apparel patterninformation, the season information, the weather information, theindoor/outdoor information, and the time information may be included inthe style characteristics. The recommendation unit may receive a stylepreference from the user or celebrity or experts as discussed above andsearch the recommendation style information matched with the receivedstyle preference, the face and the style characteristics. In addition,the style recognizer extracts style feature information from the imagesof the model/celebrity/favorite people that the user indicates directlyor indirectly through social network likes and twitter-follow profiles,and recognizes style characteristics using the extracted style featureinformation. Next, the recommender may search recommendation styleinformation matched with the face and style characteristics recognizedin the face recognizer and the style recognizer in the recommendationstyle table in which the recommendation style information is templatedaccording to face and style characteristics stored in the memory toprovide the searched recommendation style information to the user.

The method includes selecting a suitable hair-style wherein the mostsuitable hair-style can be decided based on consideration of personalityand facial features of each selector, along with adopting her preferenceand request, and at the same time to provide an image map for ahair-style where it can be easily defined what kind of an image theselected hair-style has. The system can analyze a contour of theselector's face and its image are defined for the hair-style brought byan inner line which constitutes a boundary line between a face and ahairline and an outer line which constitutes an outside of the hairstyle, and wherein it is analyzed whether the selected hair-style issuitable or not with respect to form and balance features and alsoanalyzed whether the selected hair-style is suitable to the image.Analysis for the form and the balance of the hair-style is performedbased on five elements; 1. balance between upper and lower parts of theface, 2. Silhouette, 3. Face line, 4. Balancing between head and face,and 5. Total balancing.

Further, analysis for an image of the hair-style is performed based ontwo elements; 1. An impression on the hair-style and 2. An image gapbetween face features and a hair-style. Further, with regard to theanalysis for a form of the hair-style and its balance, the analysis isperformed based on a comparison of a standard proportion between thehair and the face. Further, the system can apply predetermined standardssuch as those in U.S. Pat. No. 6,333,985 where the standard proportionbetween the hair and the face is devised as follows:

1. The placement of the eyes is in the center of the whole construction.

2. The proportion ratio between distance from the eyes to the top end ofa hair style and distance from the eyes to a bottom end of a jaw is 1:1.

3. The proportion ratio between length of a forehead, distance from thebottom of the forehead to a nose tip and distance from the nose tip tothe bottom end of the jaw is 1:1:1.

4. The proportion ratio between the length of the forehead and distancefrom the top of the forehead at the hairline to the top end of the hairstyle is 1:0.5.

5. The proportion ratio between length and breadth of the face includingthe head and hair is, 1.5:1.

Further, an image for “light” and “heavy” is prepared on a perpendicularaxis up and down, while the image for “curve line” and “straight line”is prepared on a horizontal axis, and thus representative hair-styles inaccordance with these expressed images are arranged.

Further, the image for “light” has some features considered to beyouthful such as fluffy loose hair ends, bright in color, having hairson the forehead, to appear dry, and short to medium size.

On the other hand, the image for “heavy” has features associated with tobe calm and an adult image such as stable in the hair-ends, dark incolor, no hair on the forehead, to appear wet, and medium to long hairlength.

The “curve line” depicts a warm and sweet image, with some features ofthe hair style waved and curled, a rounded line, and abundant soft hairat the sides.

The “straight line” depicts a cool and clean image, with some featuresof the style straight, an angular line, and having a long and hardsilhouette.

For the face recognizing process and the style recognizing process, therecommendation device detects a face region from a user imagetransmitted and extracts face feature information from the detected faceregion. Next, the recommendation device may recognize gender and agefrom the extracted face feature information or from the user's socialnetwork profile. In addition, the recommendation device may extractstyle feature information from a region of the user image except for theface region and recognize the user style characteristics from theextracted style feature information. The hair style information matchedwith the face characteristics is used to superimpose the hair onto theuser's 3D head model.

Then, the recommendation unit searches the recommendation styleinformation for the characteristics matched with the face and stylecharacteristics recognized in the face recognition unit and the stylerecognition unit in the recommendation style table according to thecharacteristics. Here, at least one of the hair style information,makeup style information, and recommendation product information isincluded in the recommendation style information. The recommendationunit may receive a style preference from the user and search therecommendation style information matched with the received stylepreference and the characteristics. Further, in the case in which aplurality of recommendation style information is searched, therecommendation unit may prioritize the plurality of searchedrecommendation style information according to a matched ratio with thecharacteristics.

FIG. 9A shows an exemplary process to provide trendy new hairrecommendations

Capture 3D model of head (350)

Remove current hair (352)

Select hair styles from new trends (354)

Morph or project hair onto the 3D model of head (356)

Allow user to iterative change hair styling until satisfied with newhairdo (358)

Allow user to select from a library of wardrobes to provide realisticsimulation (360)

Share desired hair style with professional to achieve desired hairdo(362)

FIG. 9B shows a corresponding process to provide furniture/appliancesuggestions, while FIG. 9C shows an exemplary UI. The process of FIG. 9Bis as follows:

Capture 3D model of products including furniture/appliance (370)

Capture target space (372)

Move or remove current product as desired (374)

Select new product and retrieve (376)

Morph or project product into the target space (378)

Allow user to iterative change product position until satisfied (380)

Allow user to select from a library of additional products to providerealistic simulation (382)

Purchase product (384)

FIGS. 9C and 9D show exemplary user interface

One embodiment uses Tango, a platform that uses computer vision to givedevices the ability to understand their position relative to the worldaround them. The Tango Tablet Development Kit is an Android device witha wide-angle camera, a depth sensing camera, accurate sensortimestamping, and a software stack that enables application developersto use motion tracking, area learning and depth sensing. This systemcreates “augmented reality,” where virtual objects not only appear to bea part of an actual environment, they can also interact with thatenvironment.

The system can search for products by space: in one embodiment the userspecifies or delineates a target volume or space and the system findsall products that fit the space. Users can stack virtual products on topof each other to help them visualize how a virtual table lamp would lookon top of a virtual table. This makes it easier than ever for customersto visualize online goods in their home at full scale, giving them anextra level of confidence when making an online purchase. This ease ofuse will further accelerate the shift from brick and mortar to online ascustomers can get a good sense of how a product would fit in their room,and what it would look like in their living space with an accurate 3Drendering of what the full-size item could look like in their home. Notonly is this a great extension of the customer experience, it's also apractical approach to figure out how the product fits into the user'sspace before ordering it.

Health Monitoring System

The 3D camera tracks movements and a 3-D scanner analyzes the viewer'sphysique. Body recognition software analyzes the body shape to determineweight loss or gain. The smart mirror can provide clothing/jewelry/hairstyling suggestions along with augmented reality view of the suggestionsso that the user can visualize the impact of the clothing or jewelry orstyling. Facial recognition software inspects the face shape todetermine health. The smart mirror can provide make-up suggestions alongwith augmented reality view of the applied suggestions so that the usercan visualize the impact of the makeup. The smart mirror can providenon-surgical body augmentation suggestions such as breast/buttockaugmentations along with augmented reality view of the body enlargementsor size reduction so that the user can visualize the impact of the bodyenhancement, along with clothing or jewelry or hair styling changes.

Built-in sensors in combination with mobile phone usage pattern andsocial network communications can detect signs of stress and othermental/emotional health states of the user. The mirrors could also becombined with other health-related apps to keep track of your caloriecount, vital signs, fitness level and sleep quality. By extrapolatingfrom the user's current behaviors, vitals and bone and muscle structure,the augmented-reality mirror can forecast the user's future health. Thecamera can measure breathing activity and/or heart rate of the user infront of the mirror or alternatively the system can bounce WiFi off thechest to detect breathing activity. The mirror highlights hard-to-seechanges in the body, such as increased fatigue, minute metabolicimbalances and more. A DNA analyzer can receive swipes from tongue, ear,and saliva, bodily fluids to capture genetic data at a high frequencyand such data can be correlated with the fitness wearable devices forsigns of health problems. Additionally, the data can be analyzed at ametropolitan level for public health purposes.

FIG. 10 shows an exemplary process to recommend cosmetic enhancementsfor women, and the process can be applied to men to improve muscularphysique appearance

Capture 3D model of user (370)

Isolate breast or butt region (372)

Model shape and size of breast or butt increase due to implant (374)

Morph or project the shape/size of breast or butt increase onto the 3Dmodel of user (376)

Allow user to iterative change breast/butt shapes/sizes until satisfiedwith new shape (378)

Allow user to select from a library of wardrobes to provide realisticsimulation (380)

Send desired shape and provide feedback to plastic surgeon to implementdesired shape and size (382)

Learning Machine

The system can identify new fashion styles by learning from the user'spreferences and/or social network information. For example the systemcan learn that the user prefers certain celebrities or friends' styleand apply this style to the user's fashion style. In some exampleembodiments, a system and method is shown for fashion matching thatinterprets a user's style or fashion and finds a match given a set offashion items. A fashion is a style of dress, while a fashion itemincludes an article of clothing, jewelry, or anything that is used todenote a style of dress. In addition to the user's existing style andwardrobe, the mirror can add potentially interesting items or styles tothe digital closet for the user by automatically retrieving styleinformation from other social networking sites such as Facebook,Twitter, Pinterest, LinkedIn, Instagram and other such social sites forthe purpose of more meaningful fashion and style match. Some examplesare, the system may retrieve and store user habits, i.e., TV shows,radio programs, songs, record albums, particular artists and actors andmovies the user likes as well as user preferences, geographicallocation, school, other network affiliation, age, likes, recentactivity, images viewed, searches etc. and much more variety ofinformation via social network APIs. To incentivize the user to uploadmore images of themselves thereby building a large database, the systemmay assign points to the user each time the user interacts with thesystem. Some examples of system interaction are: a user uploads theirown image with a new outfit, the user uploads another user's image andlabels it and links it with that user's profile name.

The system can be assisted with a number of learning machines, includingneural networks, case-based reasoning, Bayesian networks (includinghidden Markov models), or fuzzy systems. The Bayesian networks mayinclude: machine learning algorithms including—supervised learning,unsupervised learning, semi-supervised learning, reinforcement learning,transduction, learning to learn algorithms, or some other suitableBayesian network. The neural networks may include: Kohonenself-organizing network, recurrent networks, simple recurrent networks,Hopfield networks, stochastic neural networks, Boltzmann machines,modular neural networks, committee of machines, Associative NeuralNetwork (ASNN), holographic associative memory, instantaneously trainednetworks, spiking neural networks, dynamic neural networks, cascadingneural networks, neuro-fuzzy networks, or some other suitable neuralnetwork. Further, the neural networks may include: machine learningalgorithms including—supervised learning, unsupervised learning,semi-supervised learning, reinforcement learning, transduction, learningto learn algorithms, or some other suitable learning methods.

The system may use knowledge-based components such as a knowledge-basedrepository (KB). The repository may include clinical information. Forexample, it may include that “eating salt-rich food causes bloodpressure to increase.” The information may be stored in a variety offormats based on the type of inference employing them. Theknowledge-based repository may act as a repository for some or all ofthe referenced knowledge. For example, it can include reference valuesfor certain consents and variables used for inference. Accordingly, oneor more layers (e.g. a hierarchical pattern processing layer or PatternEngine) may subscribe to information from the knowledge-basedrepository. For example, one or more of the services may query theknowledge-based repository when making an inference.

In one embodiment, the knowledge-based repository may aggregate relevantclinical and/or behavioral knowledge from one or more sources. In anembodiment, one or more clinical and/or behavioral experts may manuallyspecify the required knowledge. In another embodiment, an ontology-basedapproach may be used. For example, the knowledge-based repository mayleverage the semantic web using techniques, such as statisticalrelational learning (SRL). SRL may expand probabilistic reasoning tocomplex relational domains, such as the semantic web. The SRL mayachieve this using a combination of representational formalisms (e.g.,logic and/or frame based systems with probabilistic models). Forexample, the SRL may employ Bayesian logic or Markov logic. For example,if there are two objects—‘Asian male’ and ‘smartness’, they may beconnected using the relationship ‘asian males are smart’. Thisrelationship may be given a weight (e.g., 0.3). This relationship mayvary from time to time (populations trend over years/decades). Byleveraging the knowledge in the semantic web (e.g., all references anddiscussions on the web where ‘asian male’ and ‘smartness’ are used andassociated) the degree of relationship may be interpreted from thesentiment of such references (e.g., positive sentiment: TRUE; negativesentiment: FALSE). Such sentiments and the volume of discussions maythen be transformed into weights. Accordingly, although the systemoriginally assigned a weight of 0.3, based on information from semanticweb about Asian males and smartness, may be revised to 0.9.

In an embodiment, Markov logic may be applied to the semantic web usingtwo objects: first-order formulae and their weights. The formulae may beacquired based on the semantics of the semantic web languages. In oneembodiment, the SRL may acquire the weights based on probability valuesspecified in ontologies. In another embodiment, where the ontologiescontain individuals, the individuals can be used to learn weights bygenerative learning. In some embodiments, the SRL may learn the weightsby matching and analyzing a predefined corpora of relevant objectsand/or textual resources. These techniques may be used to not only toobtain first-order waited formulae for clinical parameters, but alsogeneral information. This information may then be used when makinginferences.

For example, if the first order logic is ‘obesity causes hypertension,there are two objects involved: obesity and hypertension. If data onusers with obesity and as to whether they were diagnosed with diabetesor not is available, then the weights for this relationship may belearnt from the data. This may be extended to non-clinical examples suchas person's mood, beliefs etc.

The pattern recognizer may use the temporal dimension of data to learnrepresentations. The pattern recognizer may include a pattern storagesystem that exploits hierarchy and analytical abilities using ahierarchical network of nodes. The nodes may operate on the inputpatterns one at a time. For every input pattern, the node may provideone of three operations: 1. Storing patterns, 2. Learning transitionprobabilities, and 3. Context specific grouping.

A node may have a memory that stores patterns within the field of view.This memory may permanently store patterns and give each pattern adistinct label (e.g. a pattern number). Patterns that occur in the inputfield of view of the node may be compared with patterns that are alreadystored in the memory. If an identical pattern is not in the memory, thenthe input pattern may be added to the memory and given a distinctpattern number. The pattern number may be arbitrarily assigned and maynot reflect any properties of the pattern. In one embodiment, thepattern number may be encoded with one or more properties of thepattern.

In one embodiment, patterns may be stored in a node as rows of a matrix.In such an embodiment, C may represent a pattern memory matrix. In thepattern memory matrix, each row of C may be a different pattern. Thesedifferent patterns may be referred to as C-1, C-2, etc., depending onthe row in which the pattern is stored.

The nodes may construct and maintain a Markov graph. The Markov graphmay include vertices that correspond to the store patterns. Each vertexmay include a label of the pattern that it represents. As new patternsare added to the memory contents, the system may add new vertices to theMarkov graph. The system may also create a link between to vertices torepresent the number of transition events between the patternscorresponding to the vertices. For example, when an input pattern isfollowed by another input pattern j for the first time, a link may beintroduced between the vertices i and j and the number of transitionevents on that link may be set to 1. System may then increment thenumber of transition counts on the link from i and j whenever a patternfrom i to pattern j is observed. The system may normalize the Markovgraph such that the links estimate the probability of a transaction.Normalization may be achieved by dividing the number of transitionevents on the outgoing links of each vertex by the total number oftransition events from the vertex. This may be done for all vertices toobtain a normalized Markov graph. When normalization is completed, thesum of the transition probabilities for each node should add to 1. Thesystem may update the

In some embodiments of the present disclosure, a user or healthcareprovider may create a user profile comprising, e.g., identifying,characterizing, and/or medical information, including information abouta user's medical history, profession, and/or lifestyle. Further examplesof information that may be stored in a user profile includes diagnosticinformation such as family medical history, medical symptoms, durationof hypertension, localized vs. general hypertension, etc. Furthercontemplated as part of a user profile are non-pharmacologictreatment(s) (e.g., chiropractic, radiation, holistic, psychological,acupuncture, etc.), lifestyle characteristics (e.g., diet, alcoholintake, smoking habits), cognitive condition, behavioral health, andsocial well-being.

The methods and systems disclosed herein may rely on one or morealgorithm(s) to analyze one or more of the described metrics. Thealgorithm(s) may comprise analysis of data reported in real-time, andmay also analyze data reported in real-time in conjunction withauxiliary data stored in a hypertension management database. Suchauxiliary data may comprise, for example, historical user data such aspreviously-reported hypertension metrics (e.g., hypertension scores,functionality scores, medication use), personal medical history, and/orfamily medical history. In some embodiments, for example, the auxiliarydata includes at least one set of hypertension metrics previouslyreported and stored for a user. In some embodiments, the auxiliary dataincludes a user profile such as, e.g., the user profile described above.Auxiliary data may also include statistical data, such as hypertensionmetrics pooled for a plurality of users within a similar group orsubgroup. Further, auxiliary data may include clinical guidelines suchas guidelines relating to hypertension management, includingevidence-based clinical practice guidelines on the management of acuteand/or chronic hypertension or other chronic conditions.

Analysis of a set of hypertension metrics according to the presentdisclosure may allow for calibration of the level, degree, and/orquality of hypertension experienced by providing greater context touser-reported data. For example, associating a hypertension score of 7out of 10 with high functionality for a first user, and the same scorewith low functionality for a second user may indicate a relativelygreater debilitating effect of hypertension on the second user than thefirst user. Further, a high hypertension score reported by a user takinga particular medication such as opioid analgesics may indicate a need toadjust the user's treatment plan. Further, the methods and systemsdisclosed herein may provide a means of assessing relative changes in auser's distress due to hypertension over time. For example, ahypertension score of 5 out of 10 for a user who previously reportedconsistently lower hypertension scores, e.g., 1 out of 10, may indicatea serious issue requiring immediate medical attention.

Any combination(s) of hypertension metrics may be used for analysis inthe systems and methods disclosed. In some embodiments, for example, theset of hypertension metrics comprises at least one hypertension scoreand at least one functionality score. In other embodiments, the set ofhypertension metrics may comprise at least one hypertension score, atleast one functionality score, and medication use. More than one set ofhypertension metrics may be reported and analyzed at a given time. Forexample, a first set of hypertension metrics recording a user's currentstatus and a second set of hypertension metrics recording the user'sstatus at an earlier time may both be analyzed and may also be used togenerate one or more recommended actions.

Each hypertension metric may be given equal weight in the analysis, ormay also be given greater or less weight than other hypertension metricsincluded in the analysis. For example, a functionality score may begiven greater or less weight with respect to a hypertension score and/ormedication use. Whether and/or how to weigh a given hypertension metricmay be determined according to the characteristics or needs of aparticular user. As an example, User A reports a hypertension score of 8(on a scale of 1 to 10 where 10 is the most severe hypertension) and afunctionality score of 9 (on a scale of 1 to 10 where 10 is highestfunctioning), while User B reports a hypertension score of 8 but afunctionality score of 4. The present disclosure provides for thecollection, analysis, and reporting of this information, taking intoaccount the differential impact of one hypertension score on a user'sfunctionality versus that same hypertension score's impact on thefunctionality of a different user.

Hypertension metrics may undergo a pre-analysis before inclusion in aset of hypertension metrics and subsequent application of one or morealgorithms. For example, a raw score may be converted or scaledaccording to one or more algorithm(s) developed for a particular user.In some embodiments, for example, a non-numerical raw score may beconverted to a numerical score or otherwise quantified prior to theapplication of one or more algorithms. Users and healthcare providersmay retain access to raw data (e.g., hypertension metric data prior toany analysis)

Algorithm(s) according, to the present disclosure may analyze the set ofhypertension metrics according to any suitable methods known in the art.Analysis may comprise, for example, calculation of statistical averages,pattern recognition, application of mathematical models, factoranalysis, correlation, and/or regression analysis. Examples of analysesthat may be used herein include, but are not limited to, those disclosedin U.S. Patent Application Publication No. 2012/0246102 A1 the entiretyof which is incorporated herein by reference.

The present disclosure further provides for the determination of anaggregated hypertension assessment score. In some embodiments, forexample, a set of pairs metrics may be analyzed to generate acomprehensive and/or individualized assessment of hypertension bygenerating a composite or aggregated score. In such embodiments, theaggregated score may include a combination of at least one hypertensionscore, at least one functionality score, and medication use. Additionalmetrics may also be included in the aggregated score. Such metrics mayinclude, but are not limited to, exercise habits, mental well-being,depression, cognitive functioning, medication side effects, etc. Any ofthe aforementioned types of analyses may be used in determining anaggregated score.

The algorithm(s) may include a software program that may be availablefor download to an input device in various versions. In someembodiments, for example, the algorithm(s) may be directly downloadedthrough the Internet or other suitable communications means to providethe capability to troubleshoot a health issue in real-time. Thealgorithm(s) may also be periodically updated, e.g., provided contentchanges, and may also be made available for download to an input device.

The methods presently disclosed may provide a healthcare provider with amore complete record of a user's day-to-day status. By having access toa consistent data stream of hypertension metrics for a user, ahealthcare provider may be able to provide the user with timely adviceand real-time coaching on hypertension management options and solutions.A user may, for example, seek and/or receive feedback on hypertensionmanagement without waiting for an upcoming appointment with a healthcareprovider or scheduling a new appointment. Such real-time communicationcapability may be especially beneficial to provide users with guidanceand treatment options during intervals between appointments with ahealthcare provider. Healthcare providers may also be able to monitor auser's status between appointments to timely initiate, modify, orterminate a treatment plan as necessary. For example, a user's reportedmedication use may convey whether the user is taking too little or toomuch medication. In some embodiments, an alert may be triggered tonotify the user and/or a healthcare provider of the amount of medicationtaken, e.g., in comparison to a prescribed treatment plan. Thehealthcare provider could, for example, contact the user to discuss thetreatment plan. The methods disclosed herein may also provide ahealthcare provider with a longitudinal review of how a user responds tohypertension over time. For example, a healthcare provider may be ableto determine whether a given treatment plan adequately addresses auser's needs based on review of the user's reported hypertension metricsand analysis thereof according to the present disclosure.

Analysis of user data according to the methods presently disclosed maygenerate one or more recommended actions that may be transmitted anddisplayed on an output device. In some embodiments, the analysisrecommends that a user make no changes to his/her treatment plan orroutine. In other embodiments, the analysis generates a recommendationthat the user seek further consultation with a healthcare providerand/or establish compliance with a prescribed treatment plan. In otherembodiments, the analysis may encourage a user to seek immediate medicalattention. For example, the analysis may generate an alert to betransmitted to one or more output devices, e.g., a first output devicebelonging to the user and a second output device belonging to ahealthcare provider, indicating that the user is in need of immediatemedical treatment. In some embodiments, the analysis may not generate arecommended action. Other recommended actions consistent with thepresent disclosure may be contemplated and suitable according to thetreatment plans, needs, and/or preferences for a given user.

The present disclosure further provides a means for monitoring a user'smedication use to determine when his/her prescription will run out andrequire a refill. For example, a user profile may be created thatindicates a prescribed dosage and frequency of administration, as wellas total number of dosages provided in a single prescription. As theuser reports medication use, those hypertension metrics may betransmitted to a server and stored in a database in connection with theuser profile. The user profile stored on the database may thuscontinually update with each added metric and generate a notification toindicate when the prescription will run out based on the reportedmedication use. The notification may be transmitted and displayed on oneor more output devices, e.g., to a user and/or one or more healthcareproviders. In some embodiments, the one or more healthcare providers mayinclude a pharmacist. For example, a pharmacist may receive notificationof the anticipated date a prescription will run out in order to ensurethat the prescription may be timely refilled.

User data can be input for analysis according to the systems disclosedherein through any data-enabled device including, but not limited to,portable/mobile and stationary communication devices, andportable/mobile and stationary computing devices. Non-limiting examplesof input devices suitable for the systems disclosed herein include smartphones, cell phones, laptop computers, netbooks, personal computers(PCs), tablet PCs, fax machines, personal digital assistants, and/orpersonal medical devices. The user interface of the input device may beweb-based, such as a web page, or may also be a stand-alone application.Input devices may provide access to software applications via mobile andwireless platforms, and may also include web-based applications.

The input device may receive data by having a user, including, but notlimited to, a user, family member, friend, guardian, representative,healthcare provider, and/or caregiver, enter particular information viaa user interface, such as by typing and/or speaking. In someembodiments, a server may send a request for particular information tobe entered by the user via an input device. For example, an input devicemay prompt a user to enter sequentially a set of hypertension metrics,e.g., a hypertension score, a functionality score, and informationregarding use of one or more medications (e.g., type of medication,dosage taken, time of day, route of administration, etc.). In otherembodiments, the user may enter data into the input device without firstreceiving a prompt. For example, the user may initiate an application orweb-based software program and select an option to enter one or morehypertension metrics. In some embodiments, one or more hypertensionscales and/or functionality scales may be preselected by the applicationor software program. For example, a user may have the option ofselecting the type of hypertension scale and/or functionality scale forreporting hypertension metrics within the application or softwareprogram. In other embodiments, an application or software program maynot include preselected hypertension scales or functionality scales suchthat a user can employ any hypertension scale and/or functionality scaleof choice.

In exemplary system for mining health data for precision medicine,medical grade data from the user's physician/hospital, along with 3Dmodels, and lab test equipment data are stored in a database. Omic testequipment also generates data that is stored in another database. EHRdata from primary care physician (PHP), emergency room physicians (ER),and in-patient care data is also stored in a database. These databasesform a clinical data repository that contains medical diagnosis andtreatment information. The clinical data is high grade medicalinformation that is secured by patient privacy laws such as HIPPA. Oneexemplary process for improving healthcare using precision medicineincludes:

obtain clinical data from mirror and 3d party laboratory test equipment

obtain clinical data from one or more omic test equipment

obtain clinical data from a primary care physician database

obtain clinical data from a specialist physician database

obtain clinical data from an emergency room database

obtain clinical data from an in-patient care database

save the clinical data into a clinical data repository

obtain health data from fitness devices and from mobile phones

obtain behavioral data from social network communications and mobiledevice usage patterns

save the health data and behavioral data into a health data repositoryseparate from the clinical data repository

mine the clinical data repository and health data repository forpatients sharing similarity with the subject, including one or moresimilar biomarkers associated with health conditions

identify at least one similar health conditions and identifying one ormore corrective actions recorded in the repository and the result ofeach action for the one or more health conditions;

present the corrective action and result to the subject and recommendingan action to reduce risk from the predicted health condition

monitor the health condition using updates in the clinical datarepository and health data repository

In another embodiment for cost effective health maintenance, the systemincludes a method of insuring a subject for cancer, by:

enrolling the subject into a cost-saving program;

receiving a body sample during routine periodic examinations andcharacterizing the subject's omic information with a DNA sequencer; and

using historical omic information to detect an occurrence of a diseasesuch as cancer before the subject is suspected of having the disease;and

proactively recommending early treatments based on the omic informationreceived at each time interval to cost-effectively control disease.

Another exemplary process for applying the agents to a weight losstreatment scenario. The general goals of weight loss and management are:(1) at a minimum, to prevent further weight gain; (2) to reduce bodyweight; and (3) to maintain a lower body weight over the long term. Theinitial goal of weight loss therapy is to reduce body weight byapproximately 10 percent from baseline. If this goal is achieved,further weight loss can be attempted, if indicated through furtherevaluation. A reasonable time line for a 10 percent reduction in bodyweight is 6 months of therapy. For overweight patients with BMIs in thetypical range of 27 to 35, a decrease of 300 to 500 kcal/day will resultin weight losses of about ½ to 1 lb/week and a 10 percent loss in 6months. For more severely obese patients with BMIs>35, deficits of up to500 to 1,000 kcal/day will lead to weight losses of about 1 to 2 lb/weekand a 10 percent weight loss in 6 months. Weight loss at the rate of 1to 2 lb/week (calorie deficit of 500 to 1,000 kcal/day) commonly occursfor up to 6 months. After 6 months, the rate of weight loss usuallydeclines and weight plateaus because of a lesser energy expenditure atthe lower weight.

The agents are adaptive to the patient and allow for programmodifications based on patient responses and preferences. For example,the agent can be modified for weight reduction after age 65 to addressrisks associated with obesity treatment that are unique to older adultsor those who smoke.

The event handler can be coded to:

Receive message from patient or doctor (20)

Determine user treatment modality (22)

For each modality

Determine relevant rules (26)

For each rule

Determine responsive agent(s) (30)

For each agent

Execute agent program (34)

Get input from service provider if needed (36)

Format & send the message for the patient's mobile device (38)

The system processes a communication from a patient according to one ormore treatment scenarios. Each treatment scenario is composed of one ormore rules to be processed in a sequence that can be altered wheninvoking certain agents.

The if then rules can be described to the system using a graphical userinterface that runs on a web site, a computer, or a mobile device, andthe resulting rules are then processed by a rules engine. In oneembodiment, the if then rules are entered as a series of dropdownselectors whose possible values are automatically determined andpopulated for user selection to assist user in accurately specifying therules.

Other risk factors can be considered as rules by the agent, includingphysical inactivity and high serum triglycerides (>200 mg/dL). Whenthese factors are present, patients can be considered to haveincremental absolute risk above that estimated from the preceding riskfactors. Quantitative risk contribution is not available for these riskfactors, but their presence heightens the need for weight reduction inobese persons.

One embodiment determines high interest disease- and drug-relatedvariants in the pateint's genome and identifies top diseases with thehighest probabilities. For each disease, the system determines thepretest probability according to the patient age, gender, and ethnicity.The system then determines the independent disease-associated SNVs usedto calculate the subject's disease probability. For each disease, forexample type 2 diabetes, the system determines probability usingindependent SNVs, a likelihood ratio (LR), number of studies, cohortsizes, and the posttest probability. Blood pressure and blood glucosetrend measurements are also determined.

A patient motivation agent evaluates the following factors: reasons andmotivation for weight reduction; previous history of successful andunsuccessful weight loss attempts; family, friends, and work-sitesupport; the patient's understanding of the causes of obesity and howobesity contributes to several diseases; attitude toward physicalactivity; capacity to engage in physical activity; time availability forweight loss intervention; and financial considerations. In addition toconsidering these issues, the system can heighten a patient's motivationfor weight loss and prepare the patient for treatment through normativemessaging and warnings. This can be done by enumerating the dangersaccompanying persistent obesity and by describing the strategy forclinically assisted weight reduction. Reviewing the patients' pastattempts at weight loss and explaining how the new treatment plan willbe different can encourage patients and provide hope for successfulweight loss.

In an exemplary system for providing precision medicine, historical datafrom a large population is received and provided to a learning engine.The learning engine clusters the population into groups of similarcharacteristics and then creates a social network of patients who shareenough health/medical similarity that they are apt to share many medicalissues. Thus a user's likelihood of contracting a disease might beevaluated by knowing the disease status of other users in the sameinfluence cluster or neighborhood, whether they are closely connected tothat user or not.

The system, generally denoted by reference numeral 100, comprises one ormore central processing units CP1 . . . CPn, generally denoted byreference numeral 110. Embodiments comprising multiple processing units110 are preferably provided with a load balancing unit 115 that balancesprocessing load among the multiple processing units 110. The multipleprocessing units 110 may be implemented as separate processor componentsor as physical processor cores or virtual processors within a singlecomponent case. In a typical implementation the computer architecture100 comprises a network interface 120 for communicating with variousdata networks, which are generally denoted by reference sign DN. Thedata networks DN may include local-area networks, such as an Ethernetnetwork, and/or wide-area networks, such as the internet. In someimplementations the computer architecture may comprise a wirelessnetwork interface, generally denoted by reference numeral 125. By meansof the wireless network interface, the computer 100 may communicate withvarious access networks AN, such as cellular networks or WirelessLocal-Area Networks (WLAN). Other forms of wireless communicationsinclude short-range wireless techniques, such as Bluetooth and various“Bee” interfaces, such as XBee, ZigBee or one of their proprietaryimplementations. Depending on implementation, a user interface 140 maycomprise local input-output circuitry for a local user interface, suchas a keyboard, mouse and display (not shown). The computer architecturealso comprises memory 150 for storing program instructions, operatingparameters and variables. Reference numeral 160 denotes a program suitefor the server computer 100. Reference number 115-135 denotes anoptional interface by which the computer obtains data from externalsensors, analysis equipment or the like.

In some embodiments the data processing system is coupled with equipmentthat determines an organism's genotype from an in-vitro sample obtainedfrom the organism. In other embodiments the genotypes are determinedelsewhere and the data processing system may obtain data representativeof the genotype via any of its data interfaces.

One exemplary sensor communicating with one of the interfaces 115-135receives a biologic sample from an individual such as a bodily fluid(such as urine, saliva, plasma, or serum) or feces or a tissue sample(such as a buccal tissue sample or buccal cell). The biologic sample canthen be used to perform a genome scan. For example, DNA arrays can beused to analyze at least a portion of the genomic sequence of theindividual. Exemplary DNA arrays include GeneChip Arrays, GenFlex Tagarrays, and Genome-Wide Human SNP Array 6.0 (available from Affymetrix,Santa Clara, Calif.). In other examples, DNA sequencing withcommercially available next generation sequencing (NGS) platforms isgenerally conducted: DNA sequencing libraries are generated by clonalamplification by PCR in vitro; then the DNA is sequenced by synthesis,such that the DNA sequence is determined by the addition of nucleotidesto the complementary strand rather through chain-termination chemistry;next, the spatially segregated, amplified DNA templates are sequencedsimultaneously in a massively parallel fashion without the requirementfor a physical separation step. For microbiome analysis, cotton swabsare applied to forehead, behind ears, nose, among others, and fecalsamples are analyzed using DNA sequencing machines. In certainembodiments, whole or partial genome sequence information is used toperform the genome scans. Such sequences can be determined usingstandard sequencing methods including chain-termination (Sangerdideoxynucleotide), dye-terminator sequencing, and SOLiD™ sequencing(Applied Biosystems). Whole genome sequences can be cut by restrictionenzymes or sheared (mechanically) into shorter fragments for sequencing.DNA sequences can also be amplified using known methods such as PCR andvector-based cloning methods (e.g., Escherichia coli).

The sensors connecting to interfaces 115-135 can also include fitnesssensors such as wearable watches/clothing/shoes that monitor activity,heart rate, ECG, blood pressure, blood oxgen level, among others. Thesensors 115-135 can also detect purchase activities and on-lineactivities that reflect the user's health habits. For example, thesensors can be a data feed that picks up data relating to grocerypurchases, food expenses, restaurant spending.

In yet other examples, the sensors connecting to interfaces 115-135 canbe sensors in a phone. For example, in depression sensor, the phone candetect a person's activity and correlate to depression: people who stuckto a regular pattern of movement tended to be less depressed as peoplewith mental health problems in general have disrupted circadian rhythmsand a depressed mood may pull a user off her routine. Depressed peoplealso spends more time on their phones or browsing aimlessly, asdepressed people tend to start avoiding tasks or things they have to do,particularly when they're uncertain.

In addition to sensor captured healthcare data, healthcare data refersto any data related or relevant to a patient. Healthcare data mayinclude, but is not limited to, fitness data and healthcare-relatedfinancial data. Clinical data, as used herein, refers to any healthcareor medical data particular to a patient. In embodiments, clinical datacan be medical care or healthcare data resulting from or associated witha health or medical service performed in association with a clinician ina healthcare environment (e.g., lab test, diagnostic test, clinicalencounter, ecare, evisit, etc.). Clinical data may include, but is notlimited to, a health history of a patient, a diagnosis, a clinicianassessment, clinician narrative, a treatment, a family history(including family health history and/or family genetics), animmunization record, a medication, age, gender, date of birth,laboratory values, diagnostics, a test result, an allergy, a reaction, aprocedure performed, a social history, an advanced directive, frequencyand/or history of healthcare facility visits, current healthcareproviders and/or current healthcare provider location, preferredpharmacy, prescription benefit management data, an alert, claims data, avital, data traditionally captured at the point of care or during thecare process, a combination thereof, and the like. In the same oralternative embodiments, the clinical data may include medicalcompliance information. In certain embodiments, medical complianceinformation refers to a level of compliance of a particular patient withone or more prescribed medical treatments, such as medications, diet,physical therapy, follow up healthcare visits, and the like. In one ormore embodiments, the clinical data may include data obtained from thenatural language processing of one or more clinical assessments and/orclinical narratives.

By engaging and empowering patients to take an active role in datacollection, the footwear applies inconspicuous foot data with analyticsto improve health. One embodiment uses Google Maps to display healthactivity traffic; showing healthcare patterns based on real timereporting of anonymous data from healthcare footware devices. Healthcareorganizations can tap the power of that data to engage patients anddevelop more effective and more personalized approaches to care, therebylowering the overall cost of care.

The system identifies pre-detectable characteristics of a healthcondition, such that future incidents of the health condition may bepredicted, i.e., before the health condition occurs for diseaseprevention. One implementation includes capturing data from mobilefitness devices and establishing a plurality of health relatedcharacteristics associated with the population including walking status,weight, calorie burn. The characteristics include a plurality ofpre-detectable characteristics with a relationship between the healthrelated characteristics and at least one health condition, and analyzingat least a portion of said population in response to the relationship.

Another embodiment includes establishing at least one pre-detectablecharacteristic associated with a health condition, applying anintervention in response to the characteristic, monitoring a successcharacteristic of the intervention, and determining a cause of thesuccess characteristic.

Another embodiment builds a repository of health related characteristicsassociated with the population, the characteristics including aplurality of pre-detectable characteristics; and a processor configuredto receive the health related characteristics, establish a relationshipbetween the health related characteristics and at least one healthcondition, and analyzing at least a portion of the population inresponse to said relationship.

A population, as used herein, is any group of members. The populationmay include a high level of members, for example a group including oneor more of the five kingdoms of living things, or a subgroup, forexample a group including humans of a certain age range. The populationmay include living and/or dead members. The analysis may includepredicting a likelihood of a member developing the health condition, inresponse to the relationship. The health condition may be any type ofphysical or mental health condition, disease, and/or ailment. Inaddition, the analysis may include predicting the incidence of thehealth condition. The analysis may also include performing a simpleyes/no prediction regarding whether a member will likely develop thehealth condition. The analysis may be used to enable the management of ahealth care program, such as a program associated with a corporation, ora program offered to the public by a health care consultant or provider.If the analysis is associated with a corporation's healthcare program,the population may include some or all of the employees and retirees ofthe corporation, and associated spouses and dependents. The populationmay include other associated groups of the corporation, such asconsultants, contractors, suppliers and/or dealers. The population mayinclude participants from multiple corporations and/or the generalpublic. If the health care program is offered to the public, thepopulation may include members of the public, organizations, and/orcorporations.

The health related characteristics may include a plurality of healthcharacteristics, lifestyle characteristics and/or family healthcharacteristics associated with the members of the population. Healthcharacteristics may include characteristics indicative of a specificmember's health. For example, lifestyle characteristic may includeweight, heart rate, walking gait, sitting gait, running gait, exerciseor activity as detected by accelerometers, diet, and other factorsdetectable by fitness devices such as watches, phones, or foot sensorsdetailed above. For other example, health characteristic may includemedical characteristics (e.g., what medical visits, processes,procedures, or test have been performed associated with the member, thenumber of days the member has spent in a medical facility (e.g., ahospital), the number of visits the person has made to a doctor, etc.),drug characteristics (e.g., what type and amount of drugs are beingconsumed), a death characteristic (e.g., information associated with adeath certificate), an absenteeism characteristic, disabilitycharacteristics, characteristics associated with existing healthconditions, etc. Family health characteristics associated with themember may include information associated with the family medicalhistory of a specific member. For example, a history of a particularhealth risk within the family, e.g., heart failure, cancer, high bloodpressure, diabetes, anxiety, stress, etc. Lifestyle characteristic mayinclude a specific member's behavior characteristic(s), of which some orall may be modifiable lifestyle characteristics. A modifiable lifestylecharacteristic may include an exercise characteristic (e.g., does themember exercise, how often, what is the exercise, etc.) and/or anutrition characteristic (e.g., what types of food does the member eat,and how often). Nutrition characteristics may also include the amount ofsalt consumed during a designated period (e.g., a day), and the amountof fat and/or saturated fat consumed during a designated period. Inaddition, modifiable lifestyle characteristics may include whether themember drinks alcohol (and if so how much), a drug intakecharacteristic, (i.e., does the member take drugs, and if so how often,what kind, and how much), a weight characteristic (e.g., what does themember weigh, what is the member's desired weight, is the member on adiet, what is the member's weight indicator e.g., obese, slightlyoverweight, underweight, normal, etc.), a smoking characteristic (doesthe member smoke and if so how much), a safety characteristic (what arethe member's driving characteristics e.g., does the member where seatbelts, have one or more infractions associated with driving under theinfluence, or speeding tickets, etc.). In addition, modifiable lifestylecharacteristics may include a hypertension characteristic, a stresscharacteristic, a self-care characteristic, a self-efficacycharacteristic, a readiness to change characteristics, and aprophylactic aspirin therapy characteristic.

One method for performing population health management includesestablishing a plurality of health related characteristics associatedwith the population; establishing a relationship between the healthrelated characteristics and at least one health condition; and analyzingat least a portion of said population in response to said relationship.The system can predict a likelihood of at least one of said membersdeveloping said at least one health condition, in response to saidrelationship and/or the members health related characteristics. Thesystem can determine a prevalence of a health condition within saidpopulation in response to said health related characteristics. Theplurality of health related characteristics associated with saidpopulation can be done by establishing a plurality of self-reportedcharacteristics associated with at least a portion of said population. Aprevalence of the health condition can be determined by: establishing aplurality of claims associated with at least one os said members, saidclaims including at least one of a drug claim and a medical claim; crosschecking said plurality of claims (such as over a period of time, orover a number of tests); and establishing said prevalence in response tosaid cross checked claims. The system includes predicting a member'slikelihood of developing a condition with a stage of said condition inresponse to said prediction. The system can predict a time periodassociated with said development. The system can classify saidpopulation in response to said prediction, and then prioritize treatmentof the population in response to said prediction.

The system can recommend an intervention in response to said predictedlikelihood of development. This can be done by establishing a pluralityof intervention recommendations associated with said condition;establishing a success characteristics of said recommended intervention;establishing at least one of a readiness to change characteristic and aself-efficacy characteristic of said member; and recommending saidintervention in response to said plurality of interventionrecommendations, associated intervention success characteristics, andmember health related characteristics, said health characteristicsincluding said self-efficacy and said readiness to changecharacteristic.

The system can monitor failure/successful characteristic of saidintervention, and determining causes resulting in said successcharacteristic. The system can capture a plurality of self-reported dataassociated with at least a portion of said population having saidcondition. The self-reported data includes at least one of a lifestylecharacteristic, a family history characteristic, and a healthcharacteristic. The predictive relationship can be done by establishingat least one objective of said relationship; dynamically selecting astatistical analysis technique in response to said objective; andestablishing said relationship in response to said statistical analysistechnique. The predictive relationship can be applied to at least aportion of said population; and predicting a likelihood of developingsaid condition in response to said application.

The system can be configured to analyze the health of a populationhaving multiple members. In one embodiment, the method includes thesteps of establishing a plurality of health related characteristicsassociated with the population, the characteristics including aplurality of pre-detectable characteristics, establishing a relationshipbetween the health related characteristics and the health condition, andpredicting an incident of the health condition associated with at leastone of the members, in response to the relationship. The healthcondition may be any type of physical or mental health condition,disease, and/or ailment. For exemplary purposes the method and systemwill be discussed as they may relate to the health condition diabetes. Arepository of health related characteristics associated with apopulation may be collected. The health related characteristics may becollected through sources such as medical claims, drug claims, andself-reported information. The characteristics may include healthcharacteristics, lifestyle characteristics, and family historycharacteristics. The characteristics may include the amount of saturatedfat, unsaturated fat, fiber, salt, alcohol, cholesterol, etc. that amember consumes in a give time period. The characteristics may includeweight characteristic, such as a member's weight, BMI (Body Mass Index),abdominal girth, etc. The characteristics may also include the person'sblood pressure, standing heart rate, exercise habits (type andduration), and whether the member has hypertension. The health relatedcharacteristics of the population may be analyzed to establish theprevalence of diabetes among the population. For example, a medicalclaim having an ICD code with the prefix 250 is an indicator that themember may have diabetes. In addition, drug claims having a medicationcode descriptive of an anti-diabetes medication are indicators that themember has diabetes. The medical and/or drug claims are analyzed todetermine if two claims indicating a member may have diabetes, and thatare separated by at least three months, occur. If two claims meeting thecriteria are identified, then the member is determined to have diabetes.For example, if two separate ICD codes occur, separated by at leastthree months, or one such ICD code occurs and one drug code for antidiabetes medication occur, e.g., separated by at least three months,then the member may be determined to have diabetes.

Once the population has been analyzed to establish who has diabetes, thehistorical health related characteristics of the diabetics are then usedto establish a relationship between diabetes and the health relatedcharacteristics. For example, the health related characteristics areused to establish a neural network model, or regression model. Thetrained neural network and/or regression model will then be able topredict the likelihood a member of the population will acquire diabetes.In one embodiment, the neural network will also be able to establish whohas, or may acquire, the related diabetic characteristics of metabolicsyndrome and or glucose intolerance. Alternatively, these may be inputsto the neural network if available.

The established relationship may be reviewed to determine what thepre-detectable characteristics associated with diabetes are. Forexample, it may be determined that salt intake, consumption of saturatedfats, and alcohol consumption are three leading pre-detectablecharacteristics of acquiring diabetes. In addition, it may be determinedthat smoking is not a pre-detectable characteristic associated withdiabetes. The population may then be reviewed using the establishedrelationship. The health related characteristics of each member of thepopulation not known to have diabetes may be analyzed using therelationship. The analysis may indicate the likelihood the person willacquire diabetes (e.g., 75% likely). In addition, the pre-detectablecharacteristics associated with diabetes that are exhibited by theperson may be identified. In this manner, the likelihood of theacquiring diabetes may be established along with what pre-detectablecharacteristics are the primary contributors to this particular memberhaving diabetes.

Once the population's health related characteristics are analyzed, thepopulation may be ranked by the individual member's likelihood ofacquiring diabetes. In this manner, the type of intervention may berecommended based on the risk of acquiring diabetes, and thepre-detectable characteristics the member exhibits. In one embodiment,the interventions may be recommended by using another relationship (oran elaboration of the predictive relationship) to automatically make therecommendation based on the health related characteristics of themember, which may include the likelihood of acquiring diabetes andspecific pre-detectable characteristics exhibited, self-efficacy andreadiness to change characteristics of the member, etc. In oneembodiment, the intervention may include additional questionnaires orinterviews to acquire more specific information associated with diabetesfrom the individual. Other forms of intervention include one on onecounseling to convince the member of the seriousness of diabetes, therisk of acquiring diabetes associated with them, the ability to delay orprevent the onset of diabetes by changing specified lifestylecharacteristics, and the specific actions the member may take to modifyspecific aspects of their lifestyle associated with the pre-detectablecharacteristics. For example, if dietary issues are causing the memberto be overweight, the intervention may include, suggested changes todietary consumption, cookbooks directed towards the desired diet, oreven corporate sponsored diet counseling or involvement in a commercialdiet control program. The specific intervention recommended may be basedon the likelihood of acquiring diabetes the person has, the memberswillingness to change their diet and belief that they will be successfulin long term dietary change, and how much of a factor dietary issueswere in establishing this particular members likelihood of acquiringdiabetes.

Once the intervention recommendation is provided additional monitoringmay occur to determine if the member followed through with therecommendation (including why they did or didn't follow through),whether the intervention helped reduce the targeted characteristic(e.g., the targeted pre-detectable characteristic), and when theintervention did reduce the targeted characteristics, whether theultimate occurrence of diabetes was either delayed (which may be asubjective determination) or prevented altogether. The results of thismonitoring may then be used to update the established relationships. Inaddition, as incidents of diabetes occur, the health relatedcharacteristics of effected member may be used to further refine theestablished predictive relationship. In this manner, the health of thepopulation may be analyzed and managed relative to diabetes.

The system can receive data from electronic medical records (EMRs),activity data from patient watches and wearable devices, populationdemographic information from govt databases, consumer profileinformation from credit card companies or consumer sales companies,provider (doctor, dentist, caregiver) entered information, one or moreoutput registry databases. The EMRs may span multiple applications,multiple providers, multiple patients, multiple conditions, multiplevenues, multiple facilities, multiple organizations, and/or multiplecommunities. Embodiments of the EMRs may include one or more data storesof healthcare records, which may include one or more computers orservers that facilitate the storing and retrieval of the healthcarerecords. In some embodiments, one or more EMRs may be implemented as acloud-based platform or may be distributed across multiple physicallocations. Example embodiments of the EMRs may include hospital,ambulatory, clinic, health exchange, and health plan records systems.The EMRs may further include record systems, which store real-time ornear real-time patient (or user) information, such as wearable, bedside,or in-home patient monitors, for example. It is further contemplatedthat embodiments of the EMRs may use distinct clinical ontologies,nomenclatures, vocabularies, or encoding schemes for clinicalinformation, or clinical terms. Further, in some embodiments, the EMRsmay be affiliated with two or more separate health care entities and/orvenues that use two or more distinct nomenclatures.

In embodiments, the EMRs described herein may include healthcare data.As used herein, healthcare data refers to any healthcare or medical caredata related or relevant to a patient. Healthcare data may include, butis not limited to, clinical data and healthcare-related financial data.Clinical data, as used herein, refers to any healthcare or medical dataparticular to a patient. In embodiments, clinical data can be medicalcare or healthcare data resulting from or associated with a health ormedical service performed in association with a clinician in ahealthcare environment (e.g., lab test, diagnostic test, clinicalencounter, ecare, evisit, etc.). Clinical data may include, but is notlimited to, a health history of a patient, a diagnosis, a clinicianassessment, clinician narrative, a treatment, a family history(including family health history and/or family genetics), animmunization record, a medication, age, gender, date of birth,laboratory values, diagnostics, a test result, an allergy, a reaction, aprocedure performed, a social history, an advanced directive, frequencyand/or history of healthcare facility visits, current healthcareproviders and/or current healthcare provider location, preferredpharmacy, prescription benefit management data, an alert, claims data, avital, data traditionally captured at the point of care or during thecare process, a combination thereof, and the like. In the same oralternative embodiments, the clinical data may include medicalcompliance information. In certain embodiments, medical complianceinformation refers to a level of compliance of a particular patient withone or more prescribed medical treatments, such as medications, diet,physical therapy, follow up healthcare visits, and the like. In one ormore embodiments, the clinical data may include data obtained from thenatural language processing of one or more clinical assessments and/orclinical narratives.

In certain embodiments, healthcare-related financial data can refer toany financial information relevant to a patient, such as insurance data,claims data, payer data, etc. Such healthcare data (e.g., clinical dataand healthcare-related financial data) may be submitted by a patient, acare provider, a payer, etc. In certain embodiments where the healthcaredata is being submitted by anyone other than the patient, the patientmay be required to approve of such submission and/or may opt-in to oropt-out of having such healthcare data being submitted.

In embodiments, activity data can refer to health actions or activitiesperformed by a patient outside of, or remote from, a healthcareenvironment. Embodiments of activity data may include one or more datastores of activity data, which may include one or more computers orservers that facilitate the storing and retrieval of the activity data.In some embodiments, the activity data may be implemented as acloud-based platform or may be distributed across multiple physicallocations. Example embodiments of the activity data may includenutrition information and/or exercise information for a patient. Incertain embodiments, at least a portion of the activity data may berecorded utilizing a personal fitness tracker, a smart phone, and/or anapplication provided by a smart phone. In various embodiments, theactivity data may include data obtained from a patient's car. Forexample, in such embodiments, the activity data include data on theamount of driving the patient does versus the amount of walking thepatient does.

In one or more embodiments, the activity data may be submitted by apatient, a third party associated with a personal fitness tracker and/orsmart phone (such as a software developer or device manufacturer), acare provider, a payer, etc. In certain embodiments where the activityis being submitted by anyone other than the patient, the patient may berequired to approve of such submission and/or may opt-in to or opt-outof having such healthcare data being submitted.

The patient and/or population demographic information may include age,gender, date of birth, address, phone number, contact preferences,primary spoken language, technology access (e.g., internet, phone,computer, etc.), transportation (e.g., common modes of transportation),education level, motivation level, work status (student, full-time,retired, unemployed, etc.), and/or income. In certain embodiments, thepatient and/or population demographic information may include communityresource information, which may include, but is not limited to, fitnessfacility information, pharmacy information, food bank information,grocery store information, public assistance programs, homelessshelters, etc. In embodiments, the motivation level can include thelevel of motivation a particular patient has for maintaining theirhealth, which may be derived from other information (e.g., data frompersonal fitness tracker, indication the patient regularly visits aclinician for check-ups, consumer profile information, etc.).Embodiments of the patient and/or population demographic information mayinclude one or more data stores of demographic information which mayinclude one or more computers or servers that facilitate the storing andretrieval of the demographic information. In some embodiments, thepatient and/or population demographic information may be implemented asa cloud-based platform or may be distributed across multiple physicallocations. In embodiments, the patient and/or population demographicsmay be obtained through any source known to one skilled in the art. Forexample, in certain embodiments, at least a portion of the patientand/or population demographic information may be submitted by a thirdparty that relies on census data. In various embodiments, the patientand/or population demographic information may be obtained from more thanone source. In one embodiment, the patient may submit any or all of thepatient and/or population demographic information. In certainembodiments, all or a portion of the patient and/or populationdemographic information may be anonymized using techniques known to oneskilled in the art.

In one or more embodiments, the consumer profile information may includeany or all of the spending habits of one or more patients within apopulation. For instance, in certain embodiments, the consumer profileinformation may include information associated with grocery storepurchases, athletic or exercise equipment purchases, restaurantpurchases, and/or purchases of vitamins and/or supplements. Embodimentsof the consumer profile information may include one or more data storesof consumer profile information which may include one or more computersor servers that facilitate the storing and retrieval of the consumerprofile information. In some embodiments, the consumer profileinformation may be implemented as a cloud-based platform or may bedistributed across multiple physical locations. In one embodiment, apatient may provide the consumer profile information, for example, bylinking checking account and/or checking account purchase information toat least a portion of the population health management system and/or toa health insurance carrier.

The care provider information may include any information relating to aparticular care provider or healthcare facility. In one embodiment, thecare provider information may include information relating to the numberof healthcare providers and their specialties at a particular careprovider location. In the same or alternative embodiments, the careprovider information may include information relating to non-personneltype resources at a particular care provider location, such as theamount and types of medications and/or the amount and types of surgicalor other medical equipment. In one embodiment, the care providerinformation may include one or more of address and contact information,accepted payer information, status on accepting new patients,transactional systems, primary spoken language, hospital affiliations,and/or care delivery models. In embodiments, the care providerinformation may include information relating to the availability of anyor all resources at a particular healthcare facility including personneland/or non-personnel resources. Embodiments of the care providerinformation may include one or more data stores of care providerinformation which may include one or more computers or servers thatfacilitate the storing and retrieval of the care provider information.In some embodiments, the care provider information may be implemented asa cloud-based platform or may be distributed across multiple physicallocations. In one embodiment, the care provider information can beprovided by a healthcare provider, and/or a third party, such as aninsurance provider or management entity.

Information in the output registry databases may be categorized orclassified according to, for example, claims, diagnoses, wellness,satisfaction, population directories, and the like. In variousembodiments, each output registry may be used by, for example, ahealthcare organization to manage the health of a population segment. Inone or more embodiments, each output registry may be condition specific.By way of example, a healthcare organization or clinician may managediabetic patients within a proscribed geographic area. The condition inthis example is diabetes mellitus and the output registry may help thehealthcare organization manage a population segment with this condition.The output registry may, in one aspect, include identified patientswithin a population segment who have this condition or have risk factorsthat may lead to the development of diabetes, for example. The outputregistry may further include grouped patients within an identifiedsegment by degree of severity or risk, such as those grouped by thegrouping component of the population health server. The grouped patientsin an output registry may facilitate the generation of interventions oraction workflows designed to reduce disease severity or risk and toimprove outcome. Additional uses for the output registries are tomeasure outcomes related to treatment interventions and also toattribute patients within the identified segment to appropriatehealthcare providers (e.g., primary care physicians, care managershealth coaches, specialists such as endocrinologists, podiatrists, andthe like).

In embodiments, the plurality of EMRs may be associated with a pluralityof healthcare providers, a plurality of patients, a plurality of medicalconditions, a plurality of healthcare venues and/or facilities, aplurality of organizations, and/or a plurality of communities. Incertain embodiments, in addition to or in place of the healthcare data,the system can receive activity data from fitness devices, demographicinformation, e.g., the patient and/or population demographicinformation; consumer information, e.g., the consumer profileinformation; and provider information, e.g., the care providerinformation.

The data processed by the system of FIG. 11 is reflective of a largepopulation by including participants from diverse social, racial/ethnic,and ancestral populations living in a variety of geographies, socialenvironments, and economic circumstances, and from all age groups andhealth statuses. One embodiment applies precision medicine treatment tomany diseases, including common diseases such as diabetes, heartdisease, Alzheimer's, obesity, and mental illnesses like depression,bipolar disorder, and schizophrenia, as well as rare diseases.Importantly, the system can focus on ways to increase an individual'schances of remaining healthy throughout life.

In an implementation, social network information may be maintained in acomputer graph structure with nodes and edges such that each noderepresents a user or an organization in the network and each edgerepresents a known direct connection between two nodes. A number ofattributes described within social networks may be stored in a database,associated with each user (also referred to herein as nodes) andstrength of influence (also referred to herein as edges or distances).In some embodiments, the engine may be further configured to determinedistances to one or more of the patient members closest to a currentpatient's biological data with a diameter of at least one grouping andto indicate that the new patient is associated with the grouping basedon the comparison. In various embodiments, the engine is furtherconfigured to determine if the distance to one or more of the patientmembers closest to the new patient's filtered biological data is greaterthan a diameter of each grouping and to indicate that the new patient isnot associated with each grouping based on the comparison. The medicalcharacteristic may comprise a clinical outcome.

In one implementation, nodes may comprise attributes that include butare not limited to: a unique identifier assigned such as a user's name,address and/or other items of information; unique identifiers for thenode in each external social network containing the node, statisticalsummaries of the node's network, and pointers to the user's medicaldata. In an implementation, edges of the social network may compriseattributes that include but are not limited to the unique identifiers ofthe two nodes that are connected by the edge, the source of the node'sinformation (i.e. the external social network), the assigned socialinfluence from the first node to the second node, and the assignedsocial influence from the second node to the first node, and statisticalsummaries of the edge's contribution to the network.

The above mentioned examples are not intended to be limiting, and it isintended that any medical data is included within the scope of thisdisclosure. In an implementation, a user may be able to designate whichhealth provider sites or medical sites that may be desirable to obtaininformation from, or the sites may be automatically selected. The socialhealth content may be presented to the user or alternatively to a healthprofessional for assessment. For example, a user may be presented with alist of all of her medical connections from her health history sites. Insuch an example the user may wish to select all of the availableconnections, or may wish to limit the selection to only a certain numberof connections. A user may be asked to assign a strength of influence(for example, a numerical value) for each of the connections receivedfrom the social networks. In an implementation, the method will receiveuser influence information (data) by asking the user to assign astrength of influence for a connection that represents the user'ssimilarity to another user. Likewise, the method will receive userinfluence information (data) by asking the user to assign a strength ofinfluence for a connection that represents the medical influence thatany other user may have over the user herself. The strength of influenceinformation may be recorded into memory as an influence metric.Influence metrics may be discussed in the terms of distance, even thoughan actual distance may not exist between the points of social data usedin the method. A list of recommendations may be created for the userbase on his medical neighborhood and the behavior of others within thehealth/medical neighborhood. For example, if influential patients of theneighborhood are using and talking about certain medication or treatmentmodalities, it is likely that the user may desire to apply the samemedication/treatment. As such, a timely recommendation from a researchwould prove beneficial to both the treating professional and thepatient/user.

Exemplary systems and methods for disease management are provided. Invarious embodiments, a method comprises identifying similar patientclusters, generating groups and interconnections of the groups, eachgroup having one or more members that share medical similarities, eachinterconnection interconnecting groupings that share at least one commonmember, determining whether a new member shares medical similaritieswith the one or more members of each group and associating the newmember with one or more groups. The similarities may representsimilarities of measurements of gene expressions or similarities ofsequencing.

The system or method described herein may be deployed in part or inwhole through network infrastructures. The network infrastructure mayinclude elements such as computing devices, servers, routers, hubs,firewalls, clients, wireless communication devices, personal computers,communication devices, routing devices, and other active and passivedevices, modules or components as known in the art. The computing ornon-computing device(s) associated with the network infrastructure mayinclude, apart from other components, a storage medium such as flashmemory, buffer, stack, RAM, ROM, or the like. The processes, methods,program codes, and instructions described herein and elsewhere may beexecuted by the one or more network infrastructural elements.

What is claimed is:
 1. A method for recommending products, comprising:receiving a three-dimensional (3D) model of one or more products;performing motion tracking and understanding an environment with pointsor planes using accelerometer sensor and estimating light or color inthe environment using one video camera without a depth sensor in amobile phone; acquiring sensor data from sensors and optimizing featuresextracted from each image and sensor data, where a feature conveys dataunique to the image at a specific pixel location; and projecting theproduct in the environment.
 2. The method of claim 1, furthercomprising: iteratively changing product position until satisfied;identifying a best fitting product; setting each best fitting product'sinside dimension with dimensions from the 3D model plus a predeterminedgap; correlating different manufacturer's product sizes and creatingcorrespondences among different manufacturer products; and recommendinga new product by looking up the correspondences among differentmanufacturer products and generating an augmented or virtual realitydisplay of the new product in the environment.
 3. The method of claim 1,comprising receiving images from an infrared camera.
 4. The method ofclaim 1, wherein the product comprises an appliance or furniture.
 5. Themethod of claim 1, wherein the product comprises a wearable item, ajean, or a shirt.
 6. The method of claim 5, comprising rendering animage of the product on a mannequin.
 7. The method of claim 5,comprising monitoring user health by analyzing changes in the 3D modelover time.
 8. The method of claim 5, comprising analyzing a useranatomical portion and selecting a best fit from apparel variations. 9.The method of claim 1, wherein the product comprises cosmetic product, afacial makeup product, a hair product.
 10. The method of claim 1,comprising motion tracking, area learning and depth sensing the product.11. The method of claim 1, comprising creating a 3D model using infraredimages.
 12. The method of claim 1, comprising identifying one or morebest fitting products to the environment and displaying recommendationswith one or more best fitting products in the environment.
 13. Themethod of claim 12, wherein the best fitting products comprise clothing,shoes, cosmetics, appliances or furniture.
 14. The method of claim 1,comprising: capturing 3D model of user's feet and identifying thesubject's current best fitting shoe products; setting each best fittingshoe product's inside dimension with dimensions from the 3D model plus apredetermined gap and correlating different manufacturer's shoe sizesand creating correspondences among different manufacturer shoe products.15. A method for best fitting product variations to an environment,comprising: receiving a 3D model of a product with one or more productvariations; receiving a three-dimensional (3D) model of one or moreproducts; performing motion tracking and understanding an environmentwith points or planes using accelerometer sensor and estimating light orcolor in the environment using one video camera without a depth sensorin a mobile phone; acquiring sensor data from sensors and optimizingfeatures extracted from each image and sensor data, where a featureconveys data unique to the image at a specific pixel location; andgenerating an augmented or virtual reality display of the product in theenvironment.
 16. A method for recommending a service, comprising:receiving a model of a service to be applied to a target object;capturing a reference object with a predetermined dimension in anenvironment where the service is to be applied to the target objectusing a mobile camera; determining one more dimensions of theenvironment relative to the predetermined dimension of the referenceobject; generating a 3D model of the service as applied to the targetobject; scaling the 3D model of the generated 3D model based ondimensions of the environment and the product; and generating anaugmented or virtual reality display of the product in the environment.17. The method of claim 16, comprising applying the service to a productfrom one of: a cosmetic product, a plastic surgery medical device, afacial makeup product, a hair product.
 18. The method of claim 16,comprising: capturing images of a face and a reference object from aplurality of angles using a mobile camera; creating a 3D model of theface from the images with dimensions based on dimensions of thereference object; selecting a makeup pattern or color from a pluralityof makeup product variations; and blending the makeup pattern or coloronto the 3D model; and displaying the makeup color on the face.
 19. Themethod of claim 16, wherein the target object comprises a breastimplant, comprising recommending a breast augmentation sizing to apatient.
 20. The method of claim 16, wherein the products are facialmake-ups, comprising: capturing images of a face from a plurality ofangles using a mobile camera; detecting a skin tone from the faceimages; selecting a makeup pattern or color from a plurality of makeupproduct variations; and modeling the makeup pattern or color interactionwith the skin tone; and displaying a makeup on the face.